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Summary 

Purpose  As  decision-support  systems  (DSS)  and  associated  user  interface  (UI)  technologies 
mature  and  proliferate  into  mission-critical  applications,  and  increasingly  heterogeneous  user 
populations,  it  becomes  particularly  important  that  they  accommodate  individual  user 
characteristics.  While  some  progress  has  been  made  in  user-modeling  and  adaptive  user 
interfaces,  the  majority  of  existing  systems  continue  to  assume  normative  performance,  and  fail 
to  adapt  to  the  individual  characteristics  of  particular  users,  whether  those  that  are  relatively 
stable  over  time,  or  those  that  are  susceptible  to  situational  influences.  This  is  particularly  true 
for  adaptation  with  respect  of  the  user's  affective  state. 

Current  lack  of  accommodation  of  individual  variations  in  performance  in  most  human- 
machine  systems  can  lead  to  non-optimal  behavior  at  best,  and  critical  errors  with  disastrous 
consequences  at  worst. 

The  purpose  of  this  Phase  I  effort  was  to  develop  an  Affect  and  Belief  Adaptive  Interface 
System,  ABAIS,  capable  of  assessing  the  user’s  affective  and  belief  state  and  adapting  the  user 
interface  to  prevent  potential  performance  biases. 

Brief  Description  The  ABAIS  prototype  implements  a  four  step  adaptive  methodology 
consisting  of:  1)  sensing/inferring  the  individual's  affective  state  and  performance-relevant 
beliefs  (e.g.,  high  level  of  anxiety;  aircraft  is  under  attack);  2)  identifying  their  potential  impact 
on  performance  (e.g.,  focus  on  threatening  stimuli,  biasing  perception  towards  identification  of 
ambiguous  stimuli  as  threats);  3)  selecting  a  compensatory  strategy  (e.g.,  redirecting  focus  to 
other  salient  cues,  presentation  of  additional  information  to  reduce  ambiguity);  and  4) 
implementing  this  strategy  in  terms  of  specific  UI  adaptations  (e.g.,  highlighting  relevant  cues  or 
displays). 

The  ABAIS  architecture  consists  of  four  modules,  each  module  implementing  the 
corresponding  step  of  the  adaptive  methodology.  For  the  Phase  I  ABAIS  prototype,  we  focused 
on  a  knowledge-based  assessment  approach,  applied  to  assessment  of  anxiety  levels,  to 
demonstrate  the  feasibility  of  the  overall  adaptive  methodology.  The  knowledge-based 
assessment  approach  assumes  the  existence  of  multiple  types  of  data  (e.g.,  individual  history, 
personality,  task  context,  physiological  signals),  and  from  these  data  derives  the  likely  anxiety 
level.  Anxiety  was  selected  both  because  it  is  the  most  prevalent  affect  during  crisis  situations, 
and  because  its  influence  on  cognition  has  been  extensively  studied  and  empirical  data  exist  to 
support  specific  impact  prediction  and  adaptation  strategies. 

The  ABAIS  prototype  was  demonstrated  in  the  context  of  an  Air  Force  sweep  mission.  The 
prototype  is  integrated  with  a  flight  simulation  environment,  containing  both  an  interactive  and 
scripted  fighter  aircraft  simulation,  and  an  analyst  interaction  module.  The  latter  supports  the 
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definition  of  task  elements,  the  specification  of  knowledge  about  the  user  which  forms  the  basis 
of  the  adaptation,  and  specification  of  ABAIS  run-time  parameters.  The  key  element  of  the 
simulation,  the  pilot’s  user  interface,  consists  of  a  subset  of  the  cockpit  instruments,  a  heads-up 
display  (HUD),  radar,  and  two  windows  showing  incoming  communications  and  alarms, 
respectively. 

Findings  Development  of  the  Phase  I  ABAIS  proof-of-concept  demonstrated  feasibility  of 
the  overall  adaptive  methodology  and  its  implementation.  The  four  modules  comprising  the 
ABAIS  architecture  were  able  to  assess  the  user  state  using  a  knowledge-based  approach  and 
information  from  a  variety  of  sources  (e.g.,  static  task  context,  dynamic  external  events 
occurring  during  the  task,  pilot's  individual  history,  pilot's  personality  traits,  pilot's  training  and 
proficiency,  and  simulated  physiological  data),  predict  the  effects  of  this  state  in  the  context  of 
the  demonstration  task,  and  suggest  and  implement  specific  GUI  adaptation  strategies,  based  on 
the  pilot's  individual  information  presentation  preferences. 

Results  of  this  demonstration  indicate  the  feasibility  of  the  approach  in  general,  and  of 
knowledge-based  affect  assessment  and  GUI  adaptation  in  particular.  The  implementation  allows 
the  specification  of  highly-individualized  baseline  pilot  profiles,  which  are  used  to  infer 
individual  reactions  to  a  variety  of  specific  events,  based  on  the  pilot's  background,  personality, 
and  individual  history,  and  selected  physiological  data.  A  key  finding  was  demonstration  of  the 
feasibility  of  using  electronic  data  about  the  task  environment  as  basis  for  both  affect  and  belief 
assessment. 

We  believe  that  the  key  requirements  for  a  successful  ABAIS  system  are:  1)  limiting  the 
number,  type,  and  resolution  of  affective;  2)  using  multiple,  complementary  methods  and 
multiple  data  sources  for  affective  state  assessment;  3)  providing  individualized  user  data, 
including  details  of  past  performance,  individual  history,  personality  traits,  and  physiological 
data;  4)  constraining  the  overall  situation  in  terms  of  situation  assessment  and  behavioral 
possibilities;  5)  providing  a  wide  variety  of  task-specific  data  in  an  electronic  format;  6)  fine- 
tuning  the  rule-bases  and  inferencing  to  "personalize"  the  system  to  the  individual  user-task 
context;  and  7)  implementing  ‘benign’  adaptations,  that  is,  GUI  /  DSS  modifications  that  at  best 
enhance  and  at  worst  maintain  current  level  of  performance. 

Future  Applications  A  number  of  application  areas  exist  where  adaptation  to  affect  and 
belief  induced  biases  would  help  improve  performance,  and  where  the  implementation  of  an 
ABAIS-type  system  would  be  feasible.  These  include  both  commercial  and  government 
applications,  falling  into  three  broad  areas:  1)  real-time,  crisis-prone,  high-risk  decision-making 
environments;  2)  infotainment  and  edutainment  industry;  and  3)  training  environments  for 
emotional  and  cognitive  disabilities  and  disorders. 
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Abstract 

Currently,  the  majority  of  decision-support  systems  assume  normative  performance  and  fail 
to  adapt  to  individual  differences.  This  is  particularly  true  with  respect  to  affective  states  and 
individual  beliefs,  which  can  have  profound  impact  on  performance,  particularly  in  complex, 
crisis  situations.  We  developed  an  Affect  and  Belief  Adaptive  Interface  System  (ABAIS)  capable 
of  compensating  for  performance  biases  caused  by  users'  affective  states  and  active  beliefs.  The 
ABAIS  architecture  implements  an  adaptive  methodology  consisting  of  four  steps: 
sensing/inferring  user  affective  state  and  performance-relevant  beliefs;  identifying  their  potential 
impact  on  performance;  selecting  a  compensatoiy  strategy;  and  implementing  this  strategy  in 
terms  of  specific  GUI  adaptations.  ABAIS  provides  a  generic  adaptive  framework  for  exploring 
a  variety  of  user  state  assessment  methods  (e.g.,  knowledge-based,  self-reports,  diagnostic  tasks, 
physiological  sensing),  and  GUI  adaptation  strategies  (e.g.,  content-  and  format-based).  The 
ABAIS  performance  bias  prediction  is  based  on  existing  empirical  findings  from  emotion 
research  and  knowledge  of  specific  task  requirements.  ABAIS  was  developed  in  JAVA  and 
C++,  and  its  functionality  was  demonstrated  in  the  context  of  an  Air  Force  sweep  task.  The  focus 
of  this  Phase  I  effort  was  on  adapting  selected  cockpit  instruments  to  the  pilot’s  level  of  anxiety 
and  associated  beliefs.  Adaptation  to  a  heightened  level  of  obsessiveness  was  also  explored. 


IV 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Acknowledgments 

This  work  was  performed  under  US  Air  Force  Contract  F41624-98-C5032,  as  part  of  the 
Small  Business  Innovative  Research  (SBIR)  Program.  The  authors  wish  to  thank  the  Contracting 
Officer’s  Representative  (COR),  Dr.  Michael  McNeese  of  the  Collaborative  Systems  Technology 
Branch  of  the  Air  Force  Research  Laboratory  for  his  encouragement,  continued  technical  and 
administrative  support,  many  useful  discussions,  and  for  his  extensive  help  in  locating  subject 
matter  experts  within  the  Air  Force.  We  would  also  like  to  thank  the  following  individuals  who 
provided  valuable  information  about  a  variety  of  topics  relevant  to  this  project.  Major  Ray  King, 
Major  Joseph  Callister,  LTC  Tim  Sowin,  and  LTC  Royland  Marsh  provided  information  about 
personality  assessments  within  USAF.  Major  Daniel  Callan  and  LTC  Vincent  Caterina  provided 
useful  suggestions  and  feedback  regarding  the  task  scenario  selections,  and  Mr.  Bob  Show 
developed  the  final  demonstration  scenario  and  served  as  the  primary  subject  matter  expert.  Dr. 
Thomas  von  Wiegand  designed  and  developed  the  wireless  heart  monitor.  Mr.  Nat  Diirlach 
provided  helpful  comments  during  the  initial  design  stages  and  facilitated  the  development  of  the 
heart  monitor.  Drs.  Greg  Zacharias  and  Sandeep  Mulgund  of  Charles  River  Analytics  provided 
information  about  the  SD_PVI  system.  Drs.  Scott  Orr  and  Stefan  Hofmann  provided  information 
about  physiological  sensing.  Dr.  Rosalind  Picard  provided  information  about  wearable 
computing,  and  background  information  on  physiological  sensing  and  facial  recognition.  Mr. 
Daniel  Serfaty  of  Aptima,  Inc.  provided  information  about,  and  text  description  of,  the  DDD 
task.  Dr.  Kevin  Corker  of  San  Jose  State  University  provided  materials  about  task  allocation  and 
automation  issues. 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Table  of  Contents 

1.  Introduction . . 

1.1  The  Challenge:  Adaptive  Affective  Interfaces . 1 

1.2  The  Solution:  Affect  and  Belief  Adaptive  Interface  System . 2 

1.3  Technical  Objectives . . 

1 .4  T  echnical  Approach . . 

1.5  Summary  of  Results . . 

1.6  Report  Outline . . 

2.  Background . . 

2.1  Emotion  Research . . 

2.2  Effect  of  Emotion  on  Cognition  and  Performance . 21 

2.3  Individual  Differences  and  Personality  Research . 23 

2.3.1  Individual  Differences  Research . 23 

2.3.2  Personality  Theory  Research . 24 

2.3.2. 1  Trait  Theories . 25 

2.3. 2.2  Situationist  Models  and  Social  Learning  Theories . 27 

2. 3. 2.3  Fighter  Pilot  Personality  Assessment . 28 

2.4  Assessment  of  Affective  States . 29 

2.4. 1  Self  reports  and  Psychological  Instruments . 29 

2.4.2  Physiological  Methods . 31 

2.4.3  Facial  Recognition . . 

2.5  Situation  Awareness . 35 

2.6  Enabling  Technologies . . 

2.6.1  Wearable  Computers . 37 

2.6.3  Rule-Based  Reasoning . 41 

2.6.4  COTS  Products . . 

2.6.5  Existing  Relevant  Systems . 42 

3.  System  Description . 43 


VI 


Affect  &  Belief  Adaptive  Interface  Psychometrix  Associates 

3.1  System  Architecture  and  Functional  Overview . 43 

3.2  User  State  Assessment  Module . 47 

3.2.1  Affective  Assessment . 47 

3.2.2  Belief  /  Knowledge  Assessment . 58 

3. 2.2.1  Identifying  the  Task  Domain  Situation  Taxonomy . 58 

3. 2.2.2  Factors  Contributing  to  a  Particular  Belief  State . 60 

3.2.23  Dynamic  On-Line  Belief  Assessment . 61 

33  Impact  Prediction  Module . 64 

3.3.1  Effects  of  Affective  States  and  Personality  Traits  on  Performance . 64 

3.4  Strategy  Selection  Module . 69 

3.5  GUI  /  DSS  Adaptation  Module . 70 

3.6  ABAIS  Graphical  User  Interface  (GUI) . 76 

3.6.1  Pilot’s  GUI . 76 

3.6.2  Analyst’s  GUI . 78 

3.7  Simulation  Module . 83 

4.  System  Operation  and  Demonstration . 84 

4.1  Selection  of  Affective  States,  Personality  Characteristics,  and  Belief  States . 85 

4.2  Demonstration  Scenarios . 85 

4.2.1  Scenario  #1  -  Anxiety-Induced  Attention  and  Perception  Bias . 87 

4.2. 1.1  Key  Features  of  Scenario  #1 . 88 

4.2.2  Scenario  #2  -  Obsessiveness-Induced  Performance  Bias . 89 

4.2.2.2  Key  Features  of  Scenario  #2 . 91 

4.3  System  Demonstration:  Scenario  #1 . . 92 

4.3.1  Scenario  1  -  Frame  1  - 100  . . 101 

4.3.2  Scenario  1  -  Frame  2-  70nm . 103 

4.33  Scenario  1  -  Frame  3-50  . . 106 

4.3.4  Scenario  1  -  Frame  4-40  . . 109 

4.3.5  Scenario  1  -  Frame  5  -  37nm . 114 

4.3.6  Scenario  1  -  Frame  6-35  . . 115 

vii 


Affect  &  Belief  Adaptive  interface 


Psychometrix  Associates 


4.3.7  Scenario  1  -  Frame  7-30  nm . . 

4.3.8  Scenario  1  -  Frame  8  <  30  nm . . 

4.3.9  Scenario  1  -  Frame  9-  «  30  nm . 123 

5.  Summary,  Conclusions,  and  Recommendations . 128 

5.1  Summary . . 

5.2  Conclusions . . 

5.3  Recommendations . . 

6.  References .  134 


viii 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


List  of  Figures 

Figure  3.1-1:  ABAIS  Architecture  Implementing  the  Adaptive  Methodology . 44 

Figure  3.1-2:  Segment  of  a  Script  Matrix  Defining  the  Demonstration  Task . 46 

Figure  3.2-1:  Functionality  of  the  User  State  Assessment  Module . 47 

Figure  3.2. 1-1:  Sources  of  Information  for  Deriving  Pilot's  Affective  State . 49 

Figure  3.2. 1-2:  Rule-Based  Approach  to  Affect  Assessment . 50 

Figure  3.2.1-3:  Dialog  Boxes  for  the  Interactive  Specification  of  User  Profile  Information....  51 
Figure  3. 2. 1-4:  Example  Rules  Using  External  Events  Factors  to  Derive  User  Affective  State  52 

Figure  3.2.2. 1-1:  Belief  Assessment  Rules  Instantiated  in  the  Current  Task  Context . 60 

Figure  3.2.2.2-1:  Sources  of  Information  Used  to  Derive  Pilot's  Belief  State . 61 

Figure  3. 2.2.3- 1:  Examples  of  Cues  Used  as  Input  for  Belief  State  Assessment . 62 

Figure  3.2.23-2:  Examples  of  Belief  Assessment  Rules . 63 

Figure  33-1:  Functionality  of  the  Impact  Prediction  Module . 64 

Figure  3.4-1:  Functionality  of  the  Strategy  Selection  Module . 69 

Figure  3.5-1:  Functionality  of  the  GUI/DSS  Adaptation  Module . 71 

Figure  3.5-2:  Example  of  a  Content-Based  Adaptation . 72 

Figure  3.5-3:  Example  of  a  Format-Based  Adaptation . 73 

Figure  3.5-4:  Pilot  Information  Preference  Profile  Dialog  Box . 75 

Figure  3.5-5:  Examples  of  ABAIS  GUI  /  DSS  Adaptation  Rules . 76 

Figure  3.6. 1-1:  Screenshot  of  the  ABAIS  Pilot  GUI . 77 

Figure  3. 6.2-1:  Initial  ABAIS  Menu  Bar . 79 

Figure  3.6.2-2:  Analyst’s  GUI  Menus  Specifying  ABAIS  System  Run-Time  Parameters . 81 

Figure  3.6.2-3:  Task  Script  Definition  Window . 81 

Figure  3.6.2-4:  Background  Pilot  Information: . 81 

Figure  3.6.2-5:  Background  Pilot  Information:  Training,  Individual  History,  and  Personality  .  82 


IX 


Affect  &  Belief  Adaptive  Interface  Psychometrix  Associates 

Figure  4.3-1:  Pilot’s  GUI  at  the  Start  of  the  Simulation  Run . 100 

Figure  4.3.1-1:  Cues  Arriving  During  Frame  1  of  Simulation  Run . 101 

Figure  4.3. 1-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During  Frame  1 . 101 

Figure  4.3. 1-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During  Frame  1 . 102 

Figure  43.1-4:  Pilot’s  GUI  at  Frame  1  of  Simulation  Run . 103 

Figure  4.3.2-1:  Cues  Arriving  During  Frame  2  of  Simulation  Run. . . 104 

Figure  43.2-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During  Frame  2 . 104 

Figure  43.2-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During  Frame  2 . 105 

Figure  43.2-4:  Pilot’s  GUI  at  Frame  2  of  Simulation  Run . 106 

Figure  4.33-1:  Cues  Arriving  During  Frame  3  of  Simulation  Run . . 107 

Figure  4.33-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During  Frame  3 . 107 

Figure  4.33-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During  Frame  3 . 108 

Figure  43.3-4:  Pilot’s  GUI  at  Frame  3  of  Simulation  Run . 109 

Figure  43.4-1:  Cues  Arriving  During  Frame  4  of  Simulation  Run . 110 

Figure  43.4-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During  Frame  4 . 110 

Figure  43.4-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During  Frame  4 . Ill 

Figure  43.4-4:  Subset  of  Impact  Prediction  Rules  Instantiated  During  Frame  4 . Ill 

Figure  43.4-5:  Subset  of  Strategy  Selection  Rules  Instantiated  During  Frame  4 . 112 

Figure  43.4-6:  Pilot’s  GUI  at  Frame  4  of  Simulation  Run:  Non-Adapted  Version . 113 

Figure  43.4-7:  Pilot’s  GUI  at  Frame  4  of  Simulation  Run:  Adapted  Version . 1 14 

Figure  43.7-1:  Cues  Arriving  During  Frame  7  of  Simulation  Run . 115 

Figure  43.7-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During  Frame  7 . 116 

Figure  43.7-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During  Frame  7 . 116 

Figure  43.7-4  Subset  of  Strategy  Suggestion  Rules  Instantiated  During  Frame  7 . 117 

Figure  43.7-5:  Pilot’s  GUI  at  Frame  7  of  Simulation  Run:  Non-Adapted  Version . 118 


Affect  &  Belief  Adaptive  Interface  Psychometrix  Associates 

Figure  43.7-6:  Pilot’s  GUI  at  Frame  7  of  Simulation  Run:  Adapted  Version . 119 

Figure  43.8-1:  Cues  Arriving  During  Frame  8  of  Simulation  Run . 120 

Figure  43.8-2:  Pilot’s  Actions  During  Frame  8  of  the  Simulation  Run . 120 

Figure  43.8-4:  Subset  of  Impact  Prediction  Rules  Instantiated  During  Frame  8 . 121 

Figure  43.8-5:  Subset  of  Strategy  Selection  Rules  Instantiated  During  Frame  8 . 121 

Figure  43.8-6:  Subset  of  GUI  Adaptation  Rules  Instantiated  During  Frame  8 . 121 

Figure  43.8-7:  Pilot’s  GUI  at  Frame  8  of  Simulation  Run:  Non- Adapted  Version . 122 

Figure  43.8-8:  Pilot’s  GUI  at  Frame  8  of  Simulation  Run:  Adapted  Version . 123 

Figure  43.9-1:  Cues  Arriving  During  Frame  9  of  Simulation  Run . 124 

Figure  43.9-2:  Subset  of  Impact  Prediction  Rules  Instantiated  During  Frame  9 . 124 

Figure  43.9-3:  Subset  of  Strategy  Selection  Rules  Instantiated  During  Frame  9 . 124 

Figure  43.9-4:  Subset  of  GUI  Adaptation  Rules  Instantiated  During  Frame  9 . 125 

Figure  43.9-5:  Pilot’s  GUI  at  Frame  9  of  Simulation  Run:  Non-Adapted  Version . 126 

Figure  43.9-6:  Pilot’s  GUI  at  Frame  9  of  Simulation  Run:  Adapted  Version . 127 


XI 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


List  of  Tables 

Table  1.2-1:  Methodology  for  Proposed  Adaptive  Framework . 2 

Table  2.2-1:  Impact  of  Emotion  and  Personality  Traits  on  Cognition: . 22 

Table  2.3 .2.1-1:  Personality  Traits . 25 

Table  2.4. 1-1:  NEO-PI-R  Personality  Assessment  Scales  and  Subscales . 30 

Table  2.4. 1-2:  Aviation-Oriented  ALAPS  Personality  Assessment  Instrument . 30 

Table  3.1-1:  Methodology  for  Proposed  Adaptive  Framework . 44 

Table  3.2. 1-1:  Examples  of  Task  Specific  Factors  for  Contributing  to  Pilot’s  Affective  State  ..49 

Table  3.2. 1-2:  User  Profile  Example:  Anxiety  Prone  (High-Anxious)  Pilot  Profile . 50 

Table  3.2. 1-3:  Summary  of  Task  Context  Factors  Used  During  User  State  Assessment . 53 

Table  3.2. 1-4:  Examples  Rules  Using  Task  Context  Factors  to  Derive  User  Affective  State  . .  53 

Table  3.2. 1-5:  Summary  of  Training  and  Proficiency  Factors  for  User  State  Assessment . 54 

Table  3.2. 1-6:  Example  Rules  Using  Training  and  Proficiency  Factors . 54 

Table  3.2. 1-7:  Summary  of  Personality  Factors  for  User  State  Assessment . 55 

Table  3.2.1-8:  Example  Rule  Using  a  Personality  Factor  to  Derive  User  Affective  State . 55 

Table  3.2. 1-9:  Summary  of  Individual  History  Factors  for  User  State  Assessment . 56 

Table  3.2.1-10:  Examples  Rules  Using  Individual  History  Factors . 56 

Table  3.2. 1-11:  Summary  of  External  Events  Factors  for  User  State  Assessment . 57 

Table  3.2.1-12:  Examples  Rules  Using  External  Events  Factors  to  Derive  Affective  State  ....  57 

Table  3.2.1-13:  Examples  of  Physiological  Factors  for  User  State  Assessment . 57 

Table  3.2.2.1-1:  Examples  of  Salient  Cues  and  Possible  Situations . 59 

Table  3.2.2.3-1:  Examples  of  Rules  Deriving  Pilot's  Belief  State . 63 

Table  3.2.2.3-2:  Examples  of  Belief  Assessment  Output:  Possible  Situations . 64 

Table  3.3.1-1:  Summary  of  Generic  Effects  of  Affective  States  and  Personality  Traits  on 
Performance . 

xii 


Affect  &  Belief  Adaptive  Interface  Psychometrix  Associates 

Table  3.3. 1-2:  Example  Rules  Predicting  the  Effect  of  Anxiety  on  Performance . 67 

Table  3.4-1:  Examples  of  Generic  Rules  for  Compensatory  Strategy  Selection . 70 

Table  3.4-2:  Examples  of  Specific  Rules  for  Compensatory  Strategy  Selection . 70 

Table  3.5-1:  Riot  Information  Preference  Profile . 75 

Table  43-1:  Anxiety  Tolerant  (Low-Anxious)  Pilot  Profile . 93 

Table  4.3-2:  Anxiety  Prone  (High-Anxious)  Riot  Profile . 93 

Table  4.3-3:  Examples  of  Different  Pilot  Resentation  Reference  Profiles . 94 

Table  4.3-4:  Summary  of  Demonstration  Scenario:  Incoming  Cues  and  Pilot's  Physiological 
Signals,  Inferred  Belief  and  Affective  States  for  Low  and  High-Anxious  Pilot . 95 

Table  4.3-5:  Summary  of  Demonstration  Scenario:  Inferred  Belief  and  Affective  States  and 
Redicted  Influence,  Compensatory  Strategy,  and  GUI  /  DSS  Adaptation . 97 

Table  4.3-6:  Static  Factors  User  Assessment  Rules . 99 

Table  4.3. 1-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 102 

Table  4.3.2-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . . 105 

Table  4.33-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 108 

Table  43.4-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 112 

Table  43.5-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 114 

Table  43.6-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 115 

Table  43.7-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 117 

Table  43.8-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 122 

Table  43.9-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State . 125 


xiii 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Glossary  of  Terms 


General 

ABAIS 

AI 

CTA 

COTS 

DSS 

GUI 

KE 

SME 

UI 


Affect  and  Belief  Adaptive  Interface  System 

Artificial  Intelligence 

Cognitive  Task  Analysis 

Commercial  Off  the  Shelf 

Decision  Support  System 

Graphical  User  Interface 

Knowledge  Elicitation 

Subject  Matter  Expert 

User  Interface 


Physiological  Sensing 


BVP  Blood  Volume  Pressure 

EMG  Electromyogram 

GSR  Galvanic  Skin  Response 

AF  Sweep  Task  ^ 


BVR  Beyond  Visual  Range 

EID  Electronic  Identification 

IFF  Identification  Friend  Foe 

NCTR  Non-cooperative  target  recognition 

RWR  Radar  Warning  System 

TWS  Track  while  Scan 


XIV 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


1.  Introduction 

1.1  The  Challenge:  Adaptive  Affective  Interfaces 

As  decision-support  systems  (DSS)  and  associated  user  interface  (UI)  technologies  mature 
and  proliferate  into  mission-critical  applications,  and  increasingly  heterogeneous  user 
populations,  it  becomes  particularly  important  that  they  accommodate  individual  user 
characteristics.  While  some  progress  has  been  made  in  user-modeling  and  adaptive  user 
interfaces,  the  majority  of  existing  systems  continue  to  assume  normative  performance,  and  fail 
to  adapt  to  the  individual  characteristics  of  particular  users,  whether  those  that  are  relatively 
stable  over  time,  or  those  that  are  susceptible  to  situational  influences.  This  is  particularly  true 
for  adaptation  with  respect  of  the  user's  affective  state  and  individual,  possibly  idiosyncratic, 
beliefs. 

To  the  extent  that  adaptation  does  exist,  it  is  generally  limited  to  "novices"  vs.  "power-users" 
(Shneiderman,  1997),  and  thus  addresses  a  narrow  range  of  domain-  or  system-specific 
differences  among  users.  The  more  subtle  differences  in  skill  levels,  individual  histories  and 
beliefs,  decision-making  and  cognitive  -  styles,  not  to  mention  affective  states,  go  largely 
unacknowledged.  This  is  in  spite  of  accumulating  evidence  that  both  affective  states  and 
individual  differences  in  knowledge,  beliefs,  and  cognitive  styles  have  a  major  impact  on 
performance  (Williams  et  al.,  1997;  Eysenck,  1997;  Mineka  &  Sutton,  1992;  Isen,  1993; 
LeDoux,  1992;  Hammond  et  al.,  1987;  Deckert  et  al.,  1994;  Lussier  et  al.,  1992;  Fallesen,  1993). 
Such  lack  of  accommodation  of  individual  variations  in  performance  can  lead  to  non-optimal 
behavior  at  best,  and  critical  errors  with  disastrous  consequences  at  worst,  as  evidenced,  for 
example,  by  the  high  rate  of  fratricide  in  recent  military  operations  (Steinweg,  1995)  and  aircraft 
accidents  and  incidents,  both  civilian  and  military  (e.g.,  USS  Vincennes). 

Fortunately,  these  limitations  of  existing  systems  are  beginning  to  be  recognized  and 
addressed  in  recent  work  in  human-computer  interaction,  particularly  in  the  emerging  agent 
technologies.  There  is  increasing  interest  in  the  recognition  of,  and  accommodation  to,  individual 
differences  by  practitioners  in  the  relevant  disciplines,  including  human-computer  interaction 
(HCI),  artificial  intelligence  (Al),  and  cognitive  science.  This  is  evidenced  in  increasing  research 
efforts  in  understanding  individual  information  and  display  requirements  across  a  variety  of  tasks 
and  individual  characteristics,  ranging  from  domain  expertise,  through  cognitive  abilities  and 
individual  beliefs,  to  physical  limitations  (Ackerman  et  al.,  1989;  Michel  &  Reidel,  1988;  Egan, 
1988;  Edwards,  1995).  These  efforts  are  aided  by  a  parallel  increased  interest  in  the  study  of 
individual  differences  by  psychologists  (Revelle,  1995),  and  by  the  burgeoning  interest  in  the 
scientific  study  of  emotion  in  Al  and  cognitive  science  (Picard,  1997;  Hudlicka  &  Fellous, 
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1998),  psychology  (Williams  et  al.,  1997;  Davidson,  1992),  and  cognitive  neuroscience 
(LeDoux,  1992).  In  AI  and  cognitive  science,  this  interest  takes  a  number  of  forms,  including  the 
use  of  emotions  to  develop  more  realistic  agents  (Reilly,  1996;  Picard,  1997),  investigation  of 
basic  mechanisms  of  emotion  and  cognition  (Araujo,  1993;  Scherer,  1993),  investigations  of  how 
emotions  arise  from  an  integrated  cognitive  architecture  (Sloman,  1987),  and  models  of  how 
specific  emotions  impact  performance  (Hudlicka,  1997;  Pew  and  Mavor,  1998). 


1.2  The  Solution:  Affect  and  Belief  Adaptive  Interface  System 

To  help  address  the  existing  shortcomings  in  the  accommodation  of  individual  affective  and 
belief  states,  we  propose  to  design  and  demonstrate  an  adaptive  user  interface  (UI)  framework. 
This  framework  will  be  capable  of  adapting  both  to  the  users'  affective  state,  and  to  the  relevant 
individual  beliefs  that  might  influence  performance.  The  proposed  approach  is  based  on  a  four 
step  methodology,  summarized  in  table  1.2-1.  The  methodology  consists  of:  1)  sensing/inferring 
the  individual's  affective  state  and  performance-relevant  beliefs;  2)  identifying  their  potential 
impact  on  performance;  3)  selecting  a  compensatory  strategy;  and  4)  implementing  this  strategy 
in  terms  of  specific  UI  adaptations;  that  is,  presenting  additional  information,  or  presenting 
existing  information  in  a  format  that  facilitates  recognition  and  assimilation,  thereby  enhancing 
situation  awareness  (Endsley,  1995).  We  illustrate  this  approach  with  an  example  from  a 
frequent  Air  Force  combat  activity,  the  sweep  task,  described  below. 

Table  1.2-1:  Methodology  for  Proposed  Adaptive  Framework 


METHODOLOGY  STAGE 

EXAMPLE 

1.  ASSESS  USER  STATE 

•  Identify  affective  state 

•  Identify  individual  beliefs 

•  Affective  state:  Anxious 

•  Belief:  Result  of  individual  history  of  failures/successes 

2.  PREDICT  IMPACT 

(generic  &  task  specific) 

•  Predict  impact  of  affective  states 

•  Predict  impact  of  individual  beliefs 

Generic 

•  Anxiety  Impacts  Attention  &  Perception 

-  Increased  focus  on  threatening  stimuli 

-  Perception  of  ambiguous  signals  as  threats 

•  Indiv.  history  creates  biases  in  expectations  of  outcome 
Specific 

•  Focus  on  threat  ID  &  neglect  other  tasks 

•  Bias  towards  identifying  signals  as  threats 

•  Over-  or  underestimation  of  the  likelihood  of  failure 

•  Possible  premature  engagement  /  fratricide 

3.  SELECT  STRATEGY 

•  Select  compensatory  strategies  to 
prevent  biased  performance 

•  Present  reminders  to  prevent  neglect  of  tasks 

•  Present  broader  evidence  to  counteract  threat-estimation 
bias 

•  Present  contrary  evidence  to  counteract  failure-driven 
confirmation  bias 

4.  ADAPT  AIDING  INFO  TYPE  & 
PRESENTATION 

•  Provide  additional  data  about  ambiguous 
signals 

•  Select  info  format  enhancing  detection 
and  assimilation 

•  Direct  an  associated  DSS  to  present  additional  data  about 
ambiguous  signals 

•  Present  explicit  estimates  of  times  required  for 
task  completion  to  prevent  premature  engagement 

•  Use  multi-modal,  customized  attention-capturing 
presentation  to  assure  detection 
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Consider  a  situation  where  a  4-aircraft  F-15  wing  is  conducting  a  sweep  mission  at  night  and 
in  bad  weather.  The  lead  pilot  must  conduct  a  number  of  parallel  tasks,  including  aviating, 
navigating,  coordinating  with  the  other  pilots,  and  radar-based  search  and  visual  lookout  for 
target  enemy  aircraft  (bogeys).  As  the  wing  is  entering  the  engagement  phase,  hampered  by 
reduced  visibility  and  bad  weather,  one  of  the  engines  on  the  lead  pilot's  aircraft  fails,  thus 
further  contributing  to  his  level  of  stress  and  anxiety,  already  heightened  because  of  the  combat 
situation,  lack  of  sleep,  and  combat  fatigue. 

A  number  of  empirical  studies  provide  evidence  that  the  primary  effect  of  anxiety  is  on 
attention  and  perception,  causing  attention  to  be  focused  on  threatening  stimuli  and  causing 
perception  to  favor  interpretation  of  ambiguous  stimuli  as  threatening  (MacLeod  &  Hagan, 
1992).  In  this  case,  the  resulting  increased  anxiety  is  likely  to  cause  the  pilot  to  pay  increased 
attention  to  threatening  stimuli,  possibly  neglecting  the  other  critical  tasks  of  aviating, 
navigating,  and  coordinating  with  the  other  members  of  his  wing.  The  heightened  anxiety  also 
contributes  to  the  pilot's  tendency  to  interpret  ambiguous  signals  as  threatening,  which  may  lead 
to  misidentification  of  bogeys  or  even  fratricide.  Suppose  further,  that  the  pilot's  individual 
history  includes  several  episodes  of  similar  circumstances  where  a  failure  to  react  to  such 
ambiguous  stimuli  had  disastrous  consequences  (e.g.,  being  shot  down  by  enemy  aircraft).  The 
pilot  is  therefore  biased  from  several  sources  (affective  state  and  activated  beliefs  based  on 
individual  history)  to  identify  ambiguous  targets  as  a  bogey  and  to  enter  the  engagement  phase  of 
the  sweep  task  prematurely,  contributing  to  misidentification  of  aircraft  as  enemy  and  possible 
fratricide. 

To  compensate  for  this  potentiality,  we  envision  an  aiding  system  which  would  embed  the 
proposed  methodology  within  a  generic  affective,  adaptive  framework.  This  system  would  be 
capable  of  detecting  the  pilot's  state,  both  affective  state  and  current  predominant  beliefs  (i.e., 
increased  anxiety  and  expectations  of  failures  when  engagement  was  avoided  or  delayed), 
identifying  the  potential  impact  of  his  affective  and  belief  states  (i.e.,  possible  neglect  of  aviating 
and  coordinating  tasks,  increased  threat  estimation,  and  increased  likelihood  of  premature 
engagement),  selecting  a  compensatory  strategy  (e.g.,  presentation  of  additional  information  to 
reduce  ambiguity  or  slowing  down  decision-making  to  prevent  a  mistaken  hasty  decision),  and 
adapting  the  user  interface  by  providing  additional  information  in  a  customized  format  designed 
to  enhance  fast  recognition  and  assimilation  (i.e.,  directing  an  associated  DSS  to  collect 
additional  information  about  the  suspected  target,  suggesting  that  the  pilot  further  confer  with  the 
other  pilots  to  conduct  a  positive  identification,  guiding  the  pilot's  attention  to  additional  radar 
data  that  may  disconfirm  his  original  suspicions,  etc.). 

The  proposed  framework  will  be  able  to  accommodate  a  variety  of  complementary  methods 
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for  identifying  a  range  of  affective  states  (e.g.,  anxiety,  depression,  anger,  etc.),  and  beliefs  (e.g., 
intuitive  knowledge  gathered  from  individual  experience,  preferred  knowledge  and  procedures, 
beliefs  about  current  situation,  etc.).  Candidate  methods  will  be  discussed  in  more  detail  in 
section  3.  The  framework  will  use  knowledge-based  methods  (e.g.,  rule-based  reasoning  (RBR) 
and  case-based  reasoning  (CBR))  to  derive  the  most  likely  impact  of  the  detected  affective  and 
belief  state,  to  select  a  compensatory  strategy,  and  to  suggest  user  interface  adaptations 
implementing  the  selected  strategy. 

The  overall  objective  will  be  to  develop  a  generic  framework,  within  which  a  variety  of 
specific  methods  of  assessment,  impact  prediction,  and  adaptation  can  be  explored.  This  flexible 
design  will  then  facilitate  the  implementation  of  an  exploratory  testbed  during  Phase  II,  that 
would  support  modular  implementation  of  a  variety  of  specific  techniques  and  their  flexible 
combination  to  address  a  range  of  task  scenarios. 

For  the  purposes  of  the  Phase  I  effort,  we  limited  the  demonstration  to  the  assessment  of  the 
user's  level  of  anxiety,  via  rule-based  reasoning,  and  the  assessment  of  a  limited  set  of  beliefs 
relevant  to  outcome  estimation,  via  a  staged  process  of  domain  ontological  analysis  using  direct 
knowledge  elicitation  methods,  followed  by  on-line  dynamic  belief  assessment  using  rule-based 
reasoning.  Anxiety  was  selected  in  part  because  it  is  the  most  likely  predominant  affect  displayed 
during  crisis  situations,  and  in  part  because  its  impact  on  attention  and  perception  is  well- 
documented  from  existing  empirical  studies  (Williams  et  al.,  1997;  MacLeod  &  Hagan,  1992).  A 
follow-on  Phase  II  effort  would  then  expand  this  initially  limited  scope  to  include  additional 
affective  states  and  personality  characteristics,  additional  means  of  identifying  these  states  and 
characteristics,  and  enhanced  adaptation  strategies,  in  both  individual  and  team  settings. 

1.3  Technical  Objectives 

The  primary  objective  of  the  Phase  I  effort  was  to  assess  the  feasibility  of  the  proposed  four- 
step  affect  and  belief  adaptive  methodology.  In  the  Phase  I,  we  designed  and  developed  a 
framework  for  affective  interfaces  in  complex,  crisis  situations,  capable  of  adapting  to  a  variety 
of  user  affective  and  belief  states.  This  framework  implemented  the  adaptive  methodology 
described  in  table  1.2-1  within  a  workstation  architecture:  the  Affect  and  Belief  Adaptive 
Interface  System  (ABAIS).  ABAIS  was  demonstrated  in  the  context  of  a  sweep  task,  and 
focused  on  adapting  the  user  interface  output  to  the  pilot's  current  level  of  anxiety  and  beliefs 
resulting  from  individual  experiences.  In  the  Phase  I  effort,  we  demonstrated  a  knowledge-based 
approach  to  the  problem,  based  on  a  detailed  characterization  of  the  effects  of  a  variety  of 
affective  states,  personality  factors,  and  knowledge  schemata  on  flight  performance.  Basic 
questions  addressed  during  in  this  Phase  I  effort  included: 
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•  Impact  of  Affective  States  on  Performance:  How  do  different  affective  states 
impact  perception  and  decision-making?  How  do  the  effects  of  the  emotional  states  vary 
across  individuals  and  across  tasks?  Which  of  these  effects  are  most  relevant  in  crisis 
situations? 

•  Nature  of  Beliefs  and  their  Impact  on  Performance:  When  do  individual 
beliefs  emerge  during  performance  and  how  do  they  impact  perception  and  decision¬ 
making?  How  are  these  beliefs  related  to  different  affective  states? 

•  Methods  of  Affect  and  Belief  Assessment:  What  are  the  best  methods  for 
identifying  affective  states  and  individual  intuitive  beliefs?  Which  of  these  methods  can  be 
applied  in  real-time  and  which  are  limited  to  off-line  use?  How  can  complementary 
assessment  methods  be  combined  and  incorporated  into  an  adaptive  user  interface  to 
enhance  intelligent  aiding? 

•  Compensatory  Strategies  for  Affect-  and  Belief-Induced  Biases:  How  can 

we  distinguish  between  affect-  and  belief-induced  biases  which  are  adaptive,  from  those 
that  are  counterproductive?  What  are  the  best  ameliorative  and  compensatory  strategies  for 
counteracting  the  deleterious  impact  of  specific  affective  states  and  beliefs? 

•  Decision-Support  and  Display  Strategies:  What  specific  additional  information 
is  necessary  to  counteract  affect-  and  belief-induced  biases?  What  are  the  best  methods  of 
presenting  this  information  to  the  user  in  a  timely  and  efficient  manner?  What  specific 
displays  and  UI  methods  are  most  effective  in  counteracting  the  limitations  of  affect- 
induced  processing  and  belief-induced  cognitive  biases?  How  do  these  vary  across 
individual  preferences  and  tasks? 

•  Prototype  Fidelity  and  Full-Scope  Development:  What  level  of  prototype 
fidelity  and  scenario  complexity  are  required  to  demonstrate  the  basic  operation  of  the 
envisioned  adaptive  framework?  How  can  we  effectively  transition  the  prototype  to  a  full- 
scope  demonstrator  of  adaptive  affective  decision-aiding  in  crisis  situations? 

•  Commercial  Applications:  What  commercial  areas  are  most  likely  to  benefit  from 
this  type  of  adaptive  decision-aiding  and  what  is  the  likelihood  of  market  success  in  these 
areas?  How  can  the  ABAIS  framework  and  workstation  best  be  applied  to  a  variety  of 
federal  government  and  commercial  settings? 

By  addressing  these  questions  under  the  Phase  I  effort,  and  demonstrating  feasibility  of  the 
approach,  we  will  be  in  a  position  to  specify  detailed  requirements  for  a  Phase  II  effort  directed 
at  full-scope  development  and  testing  of  an  adaptive,  affective  interface  system. 
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1.4  Technical  Approach 

The  approach  taken  under  this  effort  focused  on  the  proof-of-concept  demonstration  of  an 
Affect  and  Belief  Adaptive  Interface  System  (ABAIS).  ABAIS  assists  the  user  (e.g.,  pilot)  in  the 
detection  of,  and  adaptation  to,  user  affective  states  (e.g.,  anxiety),  dominant  personality 
characteristics  (e.g.,  obsessiveness),  and  current  set  of  beliefs  that  may  influence  performance 
(activated  knowledge  schemata)  (e.g.,  pilot  belief  he  is  under  attack  by  hostile  aircraft),  in  an 
attempt  to  counter  the  associated  performance  biases.  Seven  specific  tasks  comprised  our  effort: 

•  Defining  a  scenario  for  ABAIS  demonstration 

•  Identifying  generic  effects  of  affective  states  and  decision-making  biases  resulting  from 
intuitive,  idiosyncratic  beliefs  from  existing  research  literature 

•  Identifying  specific  likely  effects  of  affective  states  and  examples  of  individual, 
idiosyncratic  beliefs  that  might  influence  performance  from  knowledge  elicitation 
interviews  with  expert  Air  Force  pilots  and  research  personnel 

•  Developing  knowledge  bases  for  each  of  the  four  modules  comprising  the  ABAIS  system 
architecture 

•  Designing  and  implementing  a  prototype  ABAIS  architecture 

•  Demonstrating  the  proof-of-concept  of  the  ABAIS  framework  and  prototype  in  the  context 
of  selected  test  scenarios 

•  Defining  requirements  for  Phase  II  development  and  commercialization 

The  activities  performed  under  each  of  these  tasks  are  summarized  in  the  paragraphs  below. 

We  first  defined  a  scenario  for  ABAIS  demonstration  through  a  series  of 
knowledge  elicitation  interviews  with  subject  matter  experts  (SME’s)  (USAF  and  Navy  fighter 
pilots  and  psychologists).  The  objective  was  to  define  a  scenario  which  would  provide  adequate 
complexity  to  allow  the  demonstration  of  the  adaptive  methodology  and  yet  be  feasible  to 
implement  within  the  short  time  available  for  the  Phase  I  effort.  The  scenario  had  to  provide 
opportunities  for  application  of  the  ABAIS  methodology  by  providing  situations  where  different 
affective  states,  personality  factors,  and  beliefs  could  have  varying  effects  on  situation 
assessment  and  decision  making.  The  scenario  also  had  to  be  of  sufficient  complexity  and 
difficulty  to  include  situations  likely  to  induce  varying  levels  of  anxiety.  A  fighter  pilot  context 
was  selected  to  provide  a  scenario  where  stress  levels  are  likely  to  be  high,  accurate  situation 
assessment  is  critical,  and  consequences  of  affect-  and  belief-induced  performance  biases  can  be 
catastrophic  or  at  best  expensive.  After  considering  several  options  of  tasks  and  task  contexts 
(e.g.,  instrument  failures  and  intercept  geometries  to  induce  anxiety),  we  selected  the  sweep  task 
and  a  series  of  BVR  air-to-air  combat  encounter  events  to  demonstrate  the  ABAIS  adaptive 
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methodology  in  a  real-time,  dynamic  context. 

We  then  identified  generic  effects  of  affect-  and  belief-induced 
performance  biases  by  reviewing  empirical  literature  from  four  areas:  1)  experimental, 
cognitive,  and  applied  psychology;  2)  personality  theory;  3)  human  factors;  and  4)  cognitive 
engineering.  The  findings  were  then  evaluated  for  their  potential  in  providing  information  about 
the  effects  of  affect-,  personality  trait-  and  belief-induced  biases  on  performance.  The  literature 
search  focused  both  on  the  basic  research  in  the  effects  of  affective  states  on  performance,  and 
on  the  more  applied  research  in  the  effects  of  beliefs  and  selected  affective,  personality  factors 
(e.g.,  anxiety,  obsessiveness)  on  situation  awareness  and  decision  making. 

We  then  identified  specific  likely  effects  of  affective  states,  selected 
personality  factors,  and  individual  beliefs  on  flight  performance  in  the  context 
of  the  demonstration  task  described  above.  Given  the  relative  paucity  of  empirical  studies  in  this 
area,  the  primary  means  of  identifying  these  effects  were  knowledge  elicitation  interviews  with 
pilots  and  other  aviation  subject  matter  experts  (SME's).  Part  of  this  effort  was  also  focused  on 
identifying  likely  personality  traits  and  affective  profiles  by  reviewing  existing  literature  on 
personality  testing  of  the  Air  Force  pilot  population. 

We  then  developed  knowledge  bases  for  each  of  the  four  modules  comprising  the 
ABAIS  system:  User  State  Assessment,  Impact  Prediction,  Strategy  Selection,  and  GUI/DSS 
Adaptation.  The  information  contained  in  these  knowledge  bases  was  derived  both  from  the 
literature  search  and  from  knowledge  elicitation  interviews  with  SME's,  and  focused  on 
knowledge  required  for  the  selected  demonstration  scenario.  The  rule  base  of  each  ABAIS 
module  captures  the  domain  knowledge  necessary  to  perform  each  of  the  four  steps  of  the 
adaptive  methodology.  The  rules,  and  the  associated  inference  engine,  assume  the  existence  of  a 
variety  of  data  providing  background  information  about  the  pilot  (individual  history,  personality 
characteristics,  physiological  data),  and  dynamic,  real-time  information  about  the  task  (via 
datalink  and  variety  of  on-board  detection  instruments). 

For  the  User  State  Assessment,  this  effort  consisted  first  of  identifying  the  multiple  categories 
of  factors  necessary  to  infer  the  user's  affective  state,  personality  trait,  and  predominant  beliefs, 
and  second,  of  constructing  the  rules  mapping  these  factors  onto  the  space  of  possible  affective 
and  belief  states.  For  the  Phase  I  effort  we  focused  on  anxiety,  obsessiveness,  and  beliefs 
corresponding  to  knowledge  schemata  likely  to  be  active  during  the  demonstration  task  (e.g., 
"lead  aircraft  under  attack",  "hostile  aircraft  approaching",  "Lead  at  maximum  firing  range", 
etc.).  We  reviewed  multiple  methods  of  user  state  assessment  and  selected  a  knowledge-based 
approach  for  this  Phase  I  demonstration,  as  providing  the  best  means  for  feasibility  assessment  of 
the  overall  approach. 
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For  the  Impact  Prediction  module,  this  effort  consisted  of  constructing  rules  that  mapped  the 
identified  affective  state,  personality  trait,  or  active  belief  onto  the  most  likely  effects  on  various 
elements  of  the  sweep  task  performance.  Again,  focusing  on  the  effects  of  anxiety  and 
obsessiveness,  we  mapped  these  factors  onto  their  corresponding  specific  effects  within  the 
demonstration  task  (e.g.,  anxiety  predisposes  towards  interpretation  of  "unknown"  NCTR  as 
hostile  and  narrowing  of  attention  on  highest-threat  task,  high  obsessiveness  predisposes  towards 
lengthy  echoing  of  communications  and  checking  of  instruments,  etc.).  Key  element  of  this  effort 
was  the  incorporation  of  the  reciprocal  influence  between  affect  and  belief,  with  the  potential  for 
a  performance  bias  due  to  a  positive  feedback  between  an  anxiety  state  and  an  anxiety-induced 
belief. 

For  the  Strategy  Selection  module,  we  developed  rules  mapping  the  specific  performance 
bias  (e.g.,  task  neglect,  cue  misinterpretation,  checking  behavior)  onto  a  corresponding  series  of 
compensatory  strategies  counteracting  these  biases  (e.g.,  redirecting  focus  onto  neglected  data, 
reminders  of  increasing  vulnerability  due  to  a  delayed  decision). 

Finally,  for  the  GUI  /  DSS  Adaptation  Module,  we  developed  rules  implementing  specific 
modifications  of  the  pilot  cockpit  instrument  displays  corresponding  to  the  suggested  strategy 
(e.g.,  highlighting  information  to  capture  pilot's  attention,  displaying  additional  information  to 
counter  predicted  biases,  etc.).  The  specific  GUI  modifications  also  took  into  consideration 
information  about  the  individual  pilot  preferences  for  information  presentation,  encoded  in 
customized  user  preference  profiles. 

We  then  designed  and  implemented  a  prototype  ABAIS  architecture  using 
COTS  products  and  existing  software  to  the  extent  possible.  The  architecture  consists  of  the  four 
modules  described  above  and  is  embedded  within  a  simulation  testbed  environment.  This 
environment  contains  both  an  interactive  and  scripted  fighter  aircraft  simulation,  and  an  analyst 
interaction  module.  The  latter  supports  the  definition  of  task  elements,  the  specification  of  user 
knowledge  used  to  drive  the  adaptation,  and  specification  of  ABAIS  run-time  parameters.  JAVA 
was  used  to  construct  the  ABAIS  graphical  user  interface  (GUI)  in  the  Jbuilder  development 
environment.  The  GUI  consists  of  selected  cockpit  instruments  (HUD,  radar,  and  a  notification 
window  for  alarms  etc.),  and  includes  a  variety  of  windows  showing  dialog  boxes  for  defining 
the  background  information  required  by  ABAIS  (e.g.,  pilot  preference  profile,  pilot  individual 
history),  and  dialog  boxes  for  specifying  the  system  run  time  parameters  (e.g.,  what  windows 
should  be  visible  for  monitoring  system  performance).  Charles  River  Analytics  SD_PVI  C++ 
routines  were  used  as  a  model  for  the  aircraft  flight  dynamics.  C++  was  used  to  implement  the 
overall  scenario  simulation,  the  rule-based  reasoning,  and  to  integrate  the  individual  system 
components  described  above.  The  flight  instruments  were  modeled  on  commercially-available 
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F-15  and  F-22  simulator  games. 

JAVA  was  selected  because  its  object-oriented  model  supports  rapid  prototyping  and  the 
development  of  reusable  code.  JAVA  also  has  excellent  cross-platform  portability.  Borland's 
JBuilder  2.0  was  selected  as  the  development  environment  to  support  rapid  design  and 
development  of  GUI's.  JBuilder  was  selected  because  it  provides  one  of  the  best  available  GUI 
design  environments  and  offers  a  full  suite  of  JAVA  objects.  The  original  plan  called  for 
integration  of  ABAIS  with  the  Charles  River  Analytics  SD_PVI  system  (Mulgund  et  al.,  1997). 
However,  it  was  determined  that  due  to  platform  and  software  incompatibilities,  as  well  as  the 
focus  of  the  current  effort,  the  ABAIS  demonstration  was  better  served  through  a  custom-coded 
GUI  and  simulation  environment.  The  SD_PVI  system  from  Charles  River  Analytics  thus 
served  primarily  as  a  model  for  the  flight  dynamics  required  for  the  flight  simulation.  The 
original  plan  called  for  using  JESS,  a  JAVA  version  of  the  CLIPS  rule  based  shell.  However,  it 
was  determined  that  the  complexity  of  the  Phase  I  rule  bases  did  not  justify  the  effort  required  to 
integrate  JESS  within  the  ABAIS  simulation  environment.  A  custom-coded  rule-based 
inferencing  engine  was  therefore  used  for  the  Phase  I  demonstration. 

We  designed,  constructed,  integrated,  and  evaluated  a  wireless  heart  rate 
monitor.  Although  not  originally  planned  for  the  Phase  I  effort,  we  designed  and  constructed  a 
wireless  heart  rate  monitor  as  a  means  of  prototyping  the  wearable  technology  necessary  for 
physiological  affect  assessment.  The  monitor  consists  of  a  light-weight  belt  worn  around  the 
torso  and  an  associated  wireless  transmitter.  In  our  initial  demonstration  we  integrated  the 
monitor  with  simple  ABAIS  GUI  adaptations  to  demonstrate  ABAIS'  capability  to  detect  and 
respond  to  changing  physiological  signals.  The  prototyped  system  ensemble  provides 
foundations  for  extensive  physiological  assessment  under  the  proposed  Phase  II. 

We  then  demonstrated  the  proof-of-concept  of  the  ABAIS  framework  and 

prototype  in  the  context  of  selected  task  scenarios.  ABAIS  performance  was  evaluated  by 
simulating  several  pilot  profiles,  represented  by  specific  personality,  individual  history,  and 
training  data,  in  the  context  of  the  demonstration  scenarios,  with  and  without  adaptation.  The 
different  pilot  profiles  generated  different  affective  and  belief  states  at  different  points  of  the 
scenario.  The  resulting  GUI  /  DSS  adaptations  were  derived  via  the  ABAIS  adaptive 
methodology,  using  the  rules  in  the  four  ABAIS  modules.  The  output  of  the  GUI  /  DSS 
Adaptation  recommendations  was  then  shown  on  the  corresponding  cockpit  instruments. 

The  ABAIS  implementation  supports  two  modes  of  system  operation:  1)  pilot-as-user  mode, 
where  the  user  actually  flies  the  aircraft  and  interacts  with  a  simulated  environment  consisting  of 
enemy  aircraft,  other  friendly  aircraft,  and  missiles;  and  2)  analyst-as-user,  where  the  analyst 
watches  a  scripted  task  and  monitors  the  scripted  pilot's  performance,  and  the  rule-based 
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inferencing.  These  modes  are  not  entirely  mutually  exclusive.  In  other  words,  the  analyst  and  / 
or  the  pilot  can  specify  pilot  and  system  parameters,  the  pilot  can  they  "fly"  the  aircraft,  and  both 
can  watch  the  system  output  on  simulated  cockpit  instruments  and  ABAIS  system  monitoring 
windows. 

Due  to  the  precise  coordination  required  to  fly  the  demonstration  script,  it  was  determined 
that  the  best  method  of  demonstrating  the  ABAIS  performance  under  Phase  I  would  be  in  the 
analyst-as-user  mode,  where  the  aircraft  is  controlled  by  a  script  that  emulates  the  desired  pilot 
behavior.  However,  we  did  demonstrate  the  human-in-the-loop  mode  during  the  integration  of 
the  heart  rate  monitor  with  the  ABAIS  system.  Under  this  mode  a  user  (e.g.,  pilot)  wears  the 
remote  monitor  and  the  real-time  physiological  signals  are  used  as  input  to  the  ABAIS  GUI 
adaptation  logic. 

Finally,  we  defined  requirements  for  Phase  II  full-scope  development  and 
commercialization.  To  define  specifications  for  ABAIS  in  military  environments,  we 
focused  on  identifying  further  development  and  demonstration  requirements  to  be  met  for  a  full- 
scope  affective  adaptive  user  interface  for  a  variety  of  real-time,  dynamic  environments  (e.g., 
AWACS  tasks).  We  focused  on  environments  requiring  team  coordination,  as  these  are 
becoming  increasingly  common  and  provide  a  rich  context  in  which  to  demonstrate  the  proposed 
ABAIS  methodology.  To  define  commercialization  requirements,  we  identified  promising 
commercial  market  areas  and  particular  market  segments  that  could  benefit  from  the 
development  of  a  suitably  specialized  affective,  adaptive  interface  and  decision  support  tool. 

1.5  Summary  of  Results 

The  primary  result  of  this  study  was  a  proof-of-concept  demonstration  of  an  Affect  and 
Belief  Adaptive  Interface  System  (ABAIS),  designed  to  provide  individualized  GUI  and  DSS 
adaptations  based  on  the  user's  affective  and  belief  state.  Specific  results  are  outlined  below, 
first  for  the  effort  as  a  whole,  and  then  organized  by  the  individual  subtasks  conducted. 

Overall  ABAIS  Adaptive  Methodology  Framework  Development  of  the  Phase  I 
ABAIS  proof-of-concept  demonstrated  feasibility  of  the  overall  adaptive  methodology  and  its 
implementation.  The  four  modules  comprising  the  ABAIS  architecture  were  able  to  assess  the 
user  state  using  a  knowledge-based  approach  and  information  from  a  variety  of  sources  (e.g., 
static  task  context,  dynamic  external  events  occurring  during  the  task,  pilot's  individual  history, 
pilot's  personality  traits,  pilot's  training  and  proficiency,  and  simulated  physiological  data), 
predict  the  effects  of  this  state  in  the  context  of  the  demonstration  task,  and  suggest  and 
implement  specific  GUI  adaptation  strategies  (e.g.,  modify  an  icon  or  display  to  enhance 
visibility),  based  on  the  pilot's  individual  information  presentation  preferences. 
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Results  of  this  demonstration  indicate  the  feasibility  of  the  approach  as  a  whole,  and  of 
knowledge-based  affect  assessment  and  GUI  adaptation.  The  implementation  allows  the 
specification  of  highly-individualized  baseline  pilot  profiles,  which  can  be  used  to  infer 
individual  reactions  to  a  variety  of  specific  events,  based  on  the  pilot's  background,  personality, 
and  individual  history  (i.e.,  specific  events  in  recent  past),  and  selected  physiological  data.  A  key 
finding  was  demonstration  of  the  feasibility  of  using  electronic  data  about  the  task  environment 
as  basis  for  both  affect  and  belief  assessment  (e.g.,  radar  contacts). 

Specifically,  we  believe  that  the  key  requirements  for  its  success  are:  1)  availability  of  highly 
individualized  data  about  the  system  user,  including  details  of  past  performance,  individual 
history,  personality  traits,  and  physiological  data;  2)  availability  of  a  wide  variety  of  task-specific 
data  in  an  electronic  format;  3)  use  of  multiple  methods  and  multiple  sources  of  data  for 
assessing  the  user's  affective  and  belief  state;  and  4)  ability  to  fine-tune  the  rule-bases  and 
inferencing  to  "personalize"  the  system  to  the  individual  user-task  context. 

The  specific  findings  supporting  these  conclusions,  and  the  detailed  findings  under  each  of 
the  seven  tasks  comprising  the  Phase  I  effort,  are  summarized  below. 

Scenario  Definition  A  sweep  task  scenario  was  developed  involving  two  friendly  fighter 
aircraft  engaging  several  unknown  and  presumed  hostile  aircraft  in  the  course  of  a  sweep  task. 
Multiple  variations  within  this  basic  structure  provide  opportunities  to  demonstrate  varying 
affective  states  (e.g.,  low  vs.  high  anxious),  varying  personality  traits  (e.g.,  low  vs.  high 
obsessive),  and  differences  in  individual  history  and  training  with  consequent  variations  in 
performance.  The  aircraft  instrumentation  is  loosely  modeled  on  the  F-15  fighter  aircraft  which 
is  assumed  to  be  equipped  with  a  HUD  and  datalink  connections  between  the  lead,  wingman,  and 
an  AWACS.  Critical  data  are  thus  shared  between  the  lead,  wingman  and  the  AWACS  pilots. 
The  ABAIS  system  is  demonstrated  in  the  context  of  the  lead's  cockpit  and  is  assumed  to  be 
tightly  coupled  with  the  detection  instruments  and  the  radar  and  HUD  displays.  The  ABAIS 
system  thus  has  access  to  any  data  appearing  on  the  radar,  whether  obtained  directly  from  the 
aircraft  sensors,  or  indirectly  from  wing  man  or  AWACS  via  datalink.  ABAIS  uses  these  data 
as  the  basis  for  its  dynamic  inferencing  to  assess  the  pilot's  affective  and  belief  state,  implement 
the  adaptation  strategy,  and  perform  the  GUI  adaptation. 

Effects  of  Affect,  Personality  Traits,  and  Beliefs  on  Performance  We 

reviewed  a  variety  of  literature  sources  to  identify  effects  of  affect-,  personality  trait-  and  belief- 
induced  performance  biases.  Affect  and  personality  traits:  Recent  experimental  psychology 
research  provides  increasing  evidence  for  the  existence  of  consistent  biases  in  attention, 
perception,  and  variety  of  inferencing  associated  with  different  affective  states  and  personality 
factors.  Experimental  psychology  literature  thus  represented  a  good  source  of  data  for 
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identifying  generic  effects  of  a  variety  of  affective  states  and  personality  traits.  Anxiety  was 
selected  as  the  primary  affect  and  obsessiveness  was  the  primary  personality  trait  for  the  Phase  I 
effort.  This  selection  was  made  in  part  because  much  is  known  about  both  of  these  factors  and 
their  effect  on  performance,  and  because  both  factors  can  play  a  role  in  dynamic,  real-time 
environments  of  interest  to  the  Air  Force.  The  following  effects  were  identified:  1)  effects  of 
anxiety  on  attention  (bias  towards  threat,  narrowing,  attentional  shifts)  and  perception  (bias 
towards  interpretation  of  ambiguous  signals  as  threatening);  and  2)  effects  of  obsessiveness  on 
decision  making  (delaying  of  decision  making  due  to  "checking  behaviors",  reduced  recall  of 
recent  events,  narrow  conceptual  categories).  Belief:  The  research  literature  in  human  error  and 
judgment  proved  to  be  a  good  source  of  information  for  assessing  potential  effects  of  individual 
beliefs.  Research  literature  in  the  more  applied  area  of  situation  assessment  and  situation 
awareness  also  provided  a  variety  of  data  about  the  effects  of  individual  knowledge  and  biases 
on  performance,  specifically  on  situation  assessment.  One  of  the  more  interesting  outcomes  of 
this  search  was  the  identification  of  generic  situation  assessment  and  decision  selection  biases 
that  can  result  from  active  beliefs.  These  include  confirmation  bias,  overgeneralization, 
availability,  order  effects,  internal  coherence,  consistency,  and  representativeness.  Together, 
these  findings  provided  data  about  the  generic  effects  of  affect  and  belief  on  performance, 
specifically  on  situation  assessment. 

Once  the  generic  effects  of  the  selected  factors  were  identified,  we  surveyed  the  literature  to 
identify  empirical  studies  documenting  these  effects  in  aviation-related  contexts,  specifically,  in 
the  fighter  pilot  context.  However,  given  the  relative  paucity  of  available  empirical  data,  we 
focused  instead  of  knowledge  elicitation  interviews  with  SME's  to  obtain  information  about 
possible  specific  effects  of  these  factors  in  the  context  of  the  demonstration  scenarios.  These 
interviews  provided  both  an  informal  confirmation  of  the  generic  effects  identified  in  the 
empirical  literature,  and  specific  instances  of  these  effects  in  the  context  of  the  demonstration 
task.  Namely,  instances  of  specific  effects  of  anxiety  on  attention  and  perception,  obsessiveness 
on  decision  making,  and  other  factors  such  as  aggressiveness,  impulsivity,  and  risk  taking  on 
situation  assessment  and  performance  in  general. 

We  also  reviewed  results  of  recent  studies  in  the  psychological  assessment  of  the  fighter  pilot 
population.  These  results  helped  identify  critical  personality  traits  necessary  to  define 
representative  pilot  profiles  for  the  simulation. 

User  State  Assessment  User  state  assessment  consisted  of  assessing  both  the  affective 
state  and  dominant  personality  trait,  and  identifying  the  likely  belief  state  (i.e.,  dominant 
knowledge  schemata  influencing  situation  assessment  and  subsequent  performance).  The 
affective  state  selected  for  the  Phase  I  demonstration  was  the  level  of  anxiety,  which  was 


12 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


assessed  to  be  at  one  of  several  qualitative  values  (e.g.,  low,  medium,  high),  using  a  variety  of 
background  and  real-time  data  about  the  pilot.  The  personality  trait  was  obsessiveness,  which 
was  specified  a  priori  and  assumed  to  manifest  itself  more  prominently  under  conditions  of 
heightened  anxiety.  The  belief  states  were  defined  in  terms  of  the  task  context  to  reflect  the 
pilot's  interpretation  of  the  current  dynamic  data  (e.g.,  contact  is  unknown  or  hostile)  and 
situation  assessment  (e.g.,  enemy  aircraft  are  attacking).  A  rule-based  approach  to  this  problem 
was  implemented,  which  used  information  about  the  pilot's  training,  personality,  and  individual 
history  background  specified  by  the  analyst  prior  to  a  simulation  run,  and  simulated 
physiological  data  and  dynamic  environment  data  as  generated  dynamically  during  a  simulation 
run.  The  knowledge-based  approach  to  this  problem  feasible  provided  that  the  following  criteria 
are  met:  1)  detailed  information  about  the  pilot's  individual  history,  training,  and  personality 
factors  is  available;  2)  real-time  physiological  measures  are  available;  3)  variety  of  task-relevant 
data  are  available  in  electronic  form  (e.g.,  enemy  radar  information,  enemy  aircraft  identification 
and  characteristics,  etc.);  4)  ability  to  fine-tune  and  customize  the  rule-bases  to  reflect  the 
individual  user-task  context.  In  general,  the  identification  of  a  belief  state  is  more  difficult  than 
identification  of  an  affective  state,  due  to  the  large  numbers  of  possible  knowledge  schemata. 
However,  in  highly  constrained  task  contexts  where  the  space  of  possible  cues,  situations,  and 
actions  is  limited,  belief  assessment  appears  feasible.  The  ability  to  identify  the  user's  dominant 
affective  state,  personality  trait,  and  currently  active  belief,  then  provided  input  to  the  next  stage 
of  the  adaptive  methodology:  predicting  the  influence  of  these  factors  on  performance. 

Impact  Prediction  Impact  of  the  user  state  on  performance  was  predicted  based  on 
knowledge  about  both  generic  and  specific  effects  of  anxiety  and  obsessiveness,  and  potential 
effects  specific  belief  states  (e.g.,  "aircraft  under  attack",  "hostile  aircraft  approaching",  etc.). 
This  knowledge  was  used  to  as  basis  for  a  detailed  affective  /  cognitive  task  analysis.  This 
method  of  task  analysis  focused  on  generating  a  broad  set  of  possible  behaviors  based  on  the 
collection  of  specific  performance  biases  resulting  from  the  identified  affective,  personality,  and 
belief  states.  A  rule-base  was  then  constructed  which  mapped  particular  user  states  onto  specific 
performance  biases  (e.g.,  heightened  anxiety  predisposes  towards  a  threat  estimation  bias, 
resulting  in  the  possible  misinterpretations  of  unknown  contacts  as  hostile).  A  generic  version  of 
this  approach  would  be  difficult,  in  other  words,  it  would  be  difficult  to  predict  the  effects  of 
arbitrary  beliefs  and  affective  states  on  performance  across  arbitrary  contexts.  However, 
constrained  environments,  where  information  is  available  about  both  the  task  details  and  the 
user,  provide  a  sufficiently  narrow  scope  within  which  the  necessary  space  of  possibilities  can  be 
generated  (e.g.,  targets  can  only  be  interpreted  as  unknown,  hostile,  or  friendly;  weapons  can  be 
deployed  or  not;  etc.).  The  ability  to  predict  the  influence  of  the  affective  state,  personality  trait, 
and  currently  active  belief,  on  performance  then  provided  input  to  the  next  stage  of  the  adaptive 


13 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


methodology:  selecting  a  compensatory  strategy. 

Strategy  Selection  As  with  the  impact  prediction,  the  successful  selection  of  a 
compensatory  strategy  was  possible  given  the  highly  focused  task  context,  within  which  only  a 
small  number  of  alternative  behaviors  exist  (e.g.,  focus  on  radar  vs.  focus  on  HUD;  present 
reminders  of  neglected  tasks;  remind  of  consequences  of  delayed  firing  decision,  etc.),  and  the 
assumed  availability  of  a  wide  variety  of  electronic  data  supporting  the  instantiation  of  the 
identified  strategies  (e.g.,  provide  more  evidence  for  or  against  a  particular  target  interpretation). 
The  identified  compensatory  strategy  then  provided  input  to  the  final  stage  of  the  adaptive 
methodology:  implementing  the  selected  strategy  in  terms  of  the  pilot's  GUI. 

GUI  /  DSS  Adaptation  Finally,  the  successful  GUI  adaptation  was  possible  given  the 
assumed  "glass  cockpit"  environment  and  the  possibility  of  modifying  selected  cockpit 
instrument  displays  as  needed,  to  implement  the  suggested  adaptation  strategy  (e.g.,  enhance 
icon  visibility,  present  additional  information  about  target,  display  notification  or  warning  text, 
etc.).  The  adaptation  was  further  enhanced  by  taking  into  consideration  the  pilot's  preference 
profile,  which  defined  the  pilot's  preferred  information  presentation  format.  In  this  way  the 
selected  strategy  was  optimized  for  the  individual  user. 

Design  and  Implementation  of  ABAIS  Prototype  The  COTS-based,  object- 
oriented  development  approach  produced  a  working  prototype  implementing  the  adaptive 
methodology  and  consisting  of  the  four  modules  described  above,  the  aircraft  simulation,  an 
analyst  module  supporting  interactive  specification  of  task  scripts,  pilot  information  required  for 
the  assessment,  pilot  preference  profiles,  and  the  monitoring  of  system  performance.  JAVA 
proved  to  be  a  good  environment  for  the  development  of  the  required  GUI’s  and  provided  a 
flexible  means  of  implementing  the  necessary  GUI  adaptations. 

Demonstration  of  ABAIS  ABAIS  performance  was  demonstrated  across  multiple 
sequences  of  events  in  the  context  of  the  sweep  task  scenario.  The  scenario  was  defined  in  terms 
of  a  task  script  matrix,  which  defined  the  sequence  of  dynamic  events  (e.g.,  communications 
between  lead,  wingman,  and  AWACS;  appearance  of  contacts  on  radar;  RWR  signals,  etc.).  The 
ABAIS  system  then  instantiated  these  dynamic  data,  together  with  the  background  knowledge 
about  the  pilot,  within  the  individual  module  knowledge-bases.  The  rules  in  the  individual 
modules  than  implemented  the  adaptive  methodology,  deriving  first  the  user's  anxiety  level 
(e.g.,  high-anxious)  and  dominant  belief  (e.g.,  hostile  aircraft  approaching,  aircraft  under  attack); 
predicting  their  effects  on  performance  (e.g.,  misinterpretation  of  unknown  contacts  as  hostile, 
assumption  aircraft  is  under  attack);  selecting  a  compensatory  strategy  (e.g.,  notify  pilot  of 
recent  change  in  contact  status  from  unknown  to  friendly);  and  finally  selecting  and 
implementing  a  corresponding  GUI  adaptation  (e.g.,  redirect  attention  to  radar  display,  enhance 
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recent  change  in  contact  status). 

Full-scope  Development  Based  on  the  successful  feasibility  demonstration,  we 
specified  the  requirements  for  a  full-scope  ABAIS  development  and  evaluation,  under  a  Phase  II 
design,  development,  and  evaluation  effort.  The  Phase  I  objective  was  to  establish  feasibility; 
under  Phase  II,  we  would  considerably  expand  the  scope,  increase  the  functionality  of  the 
individual  modules,  incorporate  limited  physiological  sensing  in  a  closed-loop  demonstration, 
and  fully  explore  and  evaluate  ABAIS  effectiveness  in  a  human-in-the-loop,  dynamic  team  task 
environment.  The  system  architecture  design  would  thus  follow  the  architecture  developed 
during  this  Phase  I  effort,  but  the  functionality  of  the  individual  modules  and  corresponding  rule- 
bases  would  be  considerably  expanded. 

Commercialization  Potential  We  have  identified  a  number  of  potential  market  areas 
for  the  application  of  the  adaptive  methodology  developed  during  this  Phase  I  effort  Three 
broad  areas  of  applications  were  identified:  real-time  decision  making  environments, 
characterized  by  information  overload,  decision-making  under  uncertainty,  and  high-risk, 
potential  for  crisis  situations;  training  and  therapeutic  environments,  designed  to  address 
specific  cognitive  biases,  learning  disabilities,  or  affective  disorders;  and  the  emerging 
infotainment  and  edutainment  industries. 

We  believe  that  these  results  demonstrate  the  basic  features  of  the  ABAIS  concept  for  affect 
and  belief  adaptation,  particularly  as  applied  in  a  real-time,  dynamic  setting  relevant  to  Air  Force 
operations.  The  study  was  specifically  structured  to  be  narrow  in  scope,  but  to  provide  sufficient 
depth  to  ensure  the  reliable  specification  of  requirements  for  a  full-scope  system. 

1.6  Report  Outline 

Chapter  2  provides  technical  background  on  related  research  and  current  technologies  most 
relevant  to  this  Phase  I  effort.  Section  2.1  provides  a  brief  overview  of  the  status  of  emotion 
research  and  some  key  recent  findings,  while  section  2.2  summarizes  the  effects  of  emotion  on 
cognition  and  performance.  Section  2.3  reviews  relevant  work  in  individual  differences  and 
personality  research.  Section  2.4  reviews  theory  and  methods  for  assessing  affective  states. 
Section  2.5  summarizes  research  on  situation  awareness  and  its  relevance  to  assessing  and 
adapting  to  the  user's  belief  state.  Finally,  section  2.6  summarizes  the  relevant  technologies, 
including  wearable  computing,  knowledge  elicitation  techniques,  rule-based  reasoning,  COTS 
products  for  software  development,  and  existing  systems  relevant  to  this  effort. 

Chapter  3  provides  a  description  of  the  ABAIS  prototype.  Section  3.1  defines  the  system 
architecture  and  provides  a  general  overview  of  system  functionality.  Sections  3.2  through  3.5 
then  describe  the  four  key  modules  of  this  architecture:  User  Assessment,  Impact  Prediction, 
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Strategy  Selection,  and  GUI  /  DSS  Adaptation.  Section  3.6  describes  the  ABAIS  GUI  and 
section  3.7  describes  the  simulation  environment. 

Chapter  4  demonstrates  ABAIS  functionality  by  illustrating  its  adaptation  strategies  across 
several  scenarios.  Section  4.1  provides  an  overview  of  the  adaptation  process.  Section  4.2  then 
describes  the  demonstration  scenarios  used.  Section  4.3  illustrates  the  sequence  of  steps  and  the 
ABAIS  adaptation,  via  a  series  of  screen  shots  taken  from  the  system  display. 

Chapter  5  summarizes  the  key  tasks  conducted  under  this  effort,  and  presents  the  major 
conclusions. 
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2.  Background 

This  chapter  provides  technical  background  on  existing  research  and  current  technologies 
most  relevant  to  our  effort  to  develop  an  ABAIS  concept  prototype.  Section  2.1  provides  a  brief 
overview  of  the  status  of  emotion  research  and  some  key  recent  findings,  while  section  2.2 
summarizes  the  effects  of  emotion  on  cognition  and  performance.  Section  2.3  reviews  relevant 
work  in  individual  differences  and  personality  research.  Section  2.4  reviews  theory  and  methods 
for  assessing  affective  states.  Section  2.5  summarizes  research  on  situation  awareness  and  its 
relevance  to  assessing  and  adapting  to  the  user's  belief  state.  Each  of  the  above  sections 
concludes  with  an  "Implications  for  ABAIS  system"  paragraph,  which  highlights  the  relevance 
of  the  discussed  findings  for  the  ABAIS  adaptive  methodology  and  system.  Finally,  section  2.6 
summarizes  the  relevant  technologies,  including  wearable  computing,  COTS  products  for 
software  development,  and  existing  systems  relevant  to  this  effort. 

2.1  Emotion  Research 

Although  central  to  human  development  and  functioning,  emotions  have,  until  recently,  had  a 
somewhat  marginal  status  in  both  cognitive  science  and  neuroscience.  The  study  of  emotions 
was  generally  equated  with  such  ineffable  phenomena  as  qualia  and  consciousness,  and  it  was 
not  clear  how  these  problems  could  be  addressed  or  in  what  way  the  study  of  emotions  could 
help  elucidate  the  nature  of  human  behavior  and  information  processing.  Another  problem  faced 
by  emotion  researchers  was  the  association  of  emotion  with  psychopathology  and  maladaptive 
behavior,  and  the  consequent  lack  of  acknowledgment  of  emotional  processing  as  integral  part  of 
human  performance. 

Over  the  past  10  years,  however,  important  discoveries  in  neuroscience  and  experimental 
psychology  have  contributed  to  an  interest  in  the  scientific  study  of  emotion.  A  growing  body  of 
evidence  from  neuroscience  research  points  to  the  existence  of  circuitry  processing  emotionally- 
relevant  stimuli  (i.e.,  stimuli  that  threaten  or  benefit  the  survival  of  the  organism  or  its  species) 
(LeDoux,  1989).  LeDoux  and  colleagues  have  studied  fear  conditioning  in  rats  and  identified  a 
number  of  key  results:  1)  existence  of  dedicated  circuitry  processing  stimuli  that  threaten  or 
benefit  organism  or  species  survival;  2)  evidence  that  emotional  circuitry  performs  fast,  less 
differentiated  processing  and  behavior  selection  (e.g.,  freezing  behavior  in  rats);  3)  evidence  that 
this  processing  is  mediated  by  connections  linking  sensory  organs  directly  to  emotional  circuitry 
in  the  brain,  specifically,  the  amygdala  (LeDoux,  1992).  Studies  in  experimental  psychology 
have  identified  memory  systems  with  distinct  processing  characteristics  (explicit  and  implicit 
memory)  (Schacter,  1987)  analogous  to  the  characteristics  of  the  neural  circuitry  identified  by 
LeDoux;  namely,  fast  and  less  differentiated  processing  versus  slower,  more  refined  processing. 
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Cognitive  psychologists  have  described  a  variety  of  appraisal  processes  involved  in  inducing  a 
particular  emotional  state  in  response  to  a  situation  (Frijda,  1986;  Lazarus,  1991)  and  several 
models  have  been  proposed  (Ortony  et  al.  1988),  some  of  which  have  been  implemented  in 
computational  models  (Reilly,  1996;  Scherer,  1993;  Bates  et  al.,  1992;  Frijda  and  Swagerman, 
1987).  Damasio  and  colleagues  have  studied  humans  with  brain  lesions  and  identified  the  role  of 
emotion  in  human  information  processing  and  decision-making  (Damasio,  1994).  Damasio 
suggests  that  emotions  "prune"  the  search  spaces  generated  through  cognitive  processing  by 
selecting  only  those  paths  associated  with  previously  rewarding  experiences.  Damasio's  "somatic 
marker  hypothesis"  suggests  that  this  selection  is  guided  by  "somatic  markers",  that  is,  learned 
associations  between  a  particular  affective  state  and  specific  cognitive  content. 

Recent  research  thus  provides  evidence  for  the  impact  of  emotion  on  cognitive  processing 
and  the  central  role  of  emotion  in  the  control  of  behavior.  The  emerging  findings  also  begin  to 
blur  the  distinction  between  what  has  traditionally  been  thought  of  as  the  separate  realms  of 
cognition  and  emotion. 

Specific  Roles  of  Emotions  Psychologists  and  sociologists  suggest  that  emotions  play 
both  an  intrapersonal  (intrapsychic)  and  interpersonal  (social)  roles,  functioning  as  signals  that 
communicate  the  status  of  the  organism's  goals  both  internally  and  externally  (Oatley  and 
Jenkins,  1992). 

In  the  intrapsychic  realm,  emotions  are  thought  to  be  associated  with  processing  required  to 
coordinate  activities  aimed  at  satisfying  multiple- goals  in  an  uncertain  and  unpredictable 
environment  (Oatley  and  Johnson-Laird,  1987).  Emotions  are  involved  in  the  monitoring  and 
regulation  of  goal-directed  behavior  and  are  closely  linked  with  motivation  and  preparation  for 
behavior.  Emotions  are  also  involved  in  resource  allocation,  particularly  during  high-demand 
tasks  (e.g.,  stress-producing  situations)  (Humphreys  and  Revelle,  1984).  Emotions  provide 
behavior  heuristics  by  linking  distinct  emotional  states  to  distinct  desired  behaviors  and  thereby 
function  to  improve  the  organism's  chances  for  survival  (Plutchik,  1991). 

Neuroscience  and  psychological  research  demonstrates  that  emotional  processing  is  an 
integral  part  of  adaptive  behavior  across  species.  Emotions  represent  a  phylogenetically  older 
(LeDoux,  1987),  more  primitive  yet  powerful  information  processing  mechanisms,  designed  to 
both  mediate  behaviors  which  are  particularly  adaptive  for  the  organism  and  to  make  behaviors 
more  adaptive,  by  facilitating  both  simple  reflexive  responses  and  complex  cognitive  processing. 
Recent  research  in  neuroscience  and  psychology  indicates  that  emotional  information  processing 
is  intimately  linked  with  all  functions  we  currently  understand  to  be  involved  in  cognition, 
namely  attention,  perception,  learning,  reasoning,  and  memory  storage  and  retrieval. 

Neuroscience  and  experimental  psychology  studies  of  emotional  processing  have  identified 
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the  following  functions  and  characteristics  of  emotions: 

•  Capability  to  quickly  identify  stimuli  in  the  environment  which  are  dangerous  or  beneficial 
to  the  organism's  survival.  Emotions  are  often  associated  with  hardwired  and  highly 
specific  responses  (e.g.,  rats  responding  to  squeaks  of  certain  frequencies  emitted  by  pups 
in  danger). 

•  Capability  to  induce  processing  states  which  bias  the  organism  towards  specific  types  of 
behaviors  over  long  periods  of  time. 

•  Reliance  on  pre-wired  circuitry  to  accomplish  fast  and  long  lasting  learning,  when 
necessary. 

•  Capability  to  quickly  allocate  appropriate  resources  in  critical  situations  and  thereby  focus 
attention  and  delay  less  critical  processing. 

In  the  interpersonal  realm,  behavioral  manifestations  of  emotions  serve  to  communicate 
behavioral  tendencies  (e.g.,  imminent  attack  or  withdrawal)  among  individuals  across  a  number 
of  species  and  help  coordinate  group  behaviors  and  social  interaction.  Key  to  this  is  the  ability  to 
express  emotion  on  the  one  hand,  and  recognize  others’  emotional  states  on  the  other.  This  is 
accomplished  through  a  variety  of  mechanisms,  including  posture  and  gestures,  as  well  as  facial 
expressions.  The  latter  have  been  extensively  studied  and  are  discussed  in  more  detail  in  section 
2.4.3  below. 

Basic  Emotions  The  question  of  the  existence  of  basic  emotions  addresses  the  issue  of 
how  many  distinct  “primitive”  emotions  exist  (e.g.,  fear,  anger),  which  serve  as  basis  for  the 
more  complex  emotions  (e.g.,  shame  and  guilt)  (Ekman  and  Davidson,  1994).  Much  of  emotion 
research  has  focused  on  identifying  this  set  of  basic  emotions.  While  the  exact  number  of  the 
emotions  considered  basic  varies,  the  general  agreement  is  that  there  are  between  six  and  ten 
such  basic,  or  primary,  emotions.  The  most  commonly  agreed-upon  set  includes:  fear,  anger, 
joy,  sadness,  disgust  and  surprise.  The  existence  of  basic  emotions  is  relevant  both  for 
identifying  distinct  emotional  states,  in  other  words,  identifying  methods  and  distinct 
physiological  correlates  for  these  emotions,  and  for  identifying  the  distinct  behaviors  and 
information  processing  and  performance  characteristics  of  these  emotional  states  (see  also 
discussion  section  2.2  below). 

Fundamental  Dimensions  Characterizing  Emotions  In  contrast  to  the  basic 
emotion  research  described  above,  other  efforts,  including  neurophysiological  studies,  aim  to 
identify  more  fundamental  emotional  systems  and  dimensions,  which  could  serve  to  reduce  the 
complexity  of  the  emotion  space.  These  'unified  theories  of  emotion'  typically  map  the  variety 
of  observed  or  described  emotional  states  onto  a  two  dimensional  space.  Different 
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conceptualizations  of  this  two  dimensional  space  exist.  Russel  (1978)  uses  affective  valence  and 
intensity.  Thayer  (1989)  uses  energetic  arousal  and  tense  arousal.  Watson  and  Tellegen  (1985) 
use  the  terms  positive  affect  and  negative  affect.  In  each  case,  this  2-dimnesional 
conceptualization  of  emotions  allows  locating  particular  emotional  and  mood  states  (e.g., 
sadness,  anxiety,  joy,  calm,  etc.)  within  a  simplified  space.  One  of  the  advantages  of  these  two 
dimensional  models  is  the  correspondence  between  the  distinct  dimensions  and  specific 
neurophysiological  systems  mediating  the  associated.  Empirical  evidence  suggests  that  two 
separate  neurophysiological  systems  exist  mediating  positive  and  negative  affect,  and  there  is 
evidence  for  two  independent  systems  mediating  approach  and  avoidance  behaviors  (behavioral 
inhibition  system  or  BIS  and  behavioral  approach  system  or  BAS). 

Implications  for  ABAIS  Aside  from  the  motivation  to  understand  emotional 
processing  for  its  own  sake,  as  an  integral  component  of  human  information  processing,  studying 
emotion  and  its  impact  on  information  processing  and  behavior  has  wider  implications. 
Specifically,  in  cognitive  engineering  and  human-centered  system  design,  understanding  the 
impact  of  emotions  on  cognition  and  behavior  can  help  design  better  decision-support  and 
training  systems,  by  adapting  the  system  functionality  and  user  interface  to  a  variety  of 
emotional  states,  and  by  helping  to  counteract  emotion-induced  cognitive  biases.  In  team 
research  and  applications,  understanding  the  emotion-mediated  coordination  behaviors,  and  the 
effect  of  emotion  on  team  performance  can  significantly  enhance  team  effectiveness.  In 
intelligent  agent  research,  understanding  the  role  of  emotions  in  motivation  and  communication 
can  help  produce  more  believable  agents,  more  efficient  means  of  coordination  among  multiple 
agents,  and  improved  adaptive  systems  in  general. 

Examples  of  specific  implications  of  this  research  for  ABAIS  include: 

•  Basic  emotions  imply  different  behaviors,  therefore  different  effects  on  performance, 
therefore  different  adaptations  required  to  compensate  for  potential  biases. 

•  Basic  emotions  suggests  how  many  and  which  emotions  ABAIS  should  address. 

•  Resource  allocation  research  in  emotion  helps  identify  specific  performance  influences  and 
effects  of  emotional  states,  and  suggests  compensatory  strategies. 

•  Two-dimensional  conceptualizations  of  emotion,  and  corresponding  physiological 
mechanisms,  provide  basis  for  physiological  assessment  of  the  basic  affective  components 
of  behavior  (arousal  and  valence). 

•  Interpersonal,  communication  goals  of  emotions  are  particularly  critical  in  team 
environments,  and  provide  an  additional  important  application  of  the  ABAIS  methodology 
in  team  settings. 
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2.2  Effect  of  Emotion  on  Cognition  and  Performance 

Cognitive  and  clinical  psychologists  have  observed  the  differential  impact  of  various 
emotional  states  on  cognition;  for  example,  increased  attention  to  threatening  stimuli  in  states  of 
anxiety,  increased  elaboration  of  material  in  positive  affective  states,  and  the  general 
phenomenon  of  mood-congruent  recall  (Williams  et  al.,  1997;  Mineka  and  Sutton,  1992;  Bower, 
1981;  Blaney,  1986;  Isen,  1993).  A  number  of  affective  states  and  personality  traits  have  been 
studied  extensively:  anxiety,  and  depression2,  obsessiveness,  extraversion,  and  impulsivity. 
These  factors  influence  a  number  of  perceptual  and  cognitive  processes,  including  attention, 
perceptual  categorization,  memory,  and  general  inferencing  and  judgment.  The  findings  of  these 
studies  are  briefly  summarized  below  and  in  table  2.2-1. 

Anxiety  The  primary  impact  of  anxiety  is  on  attention.  Specifically,  anxiety  narrows  the 
focus  of  attention,  predisposes  towards  the  detection  of  threatening  stimuli,  and  predisposes 
towards  the  interpretation  of  ambiguous  stimuli  as  dangerous  (Williams  et  al.,  1997;  Mineka  and 
Sutton,  1992). 

Depression  The  primary  impact  of  depression  is  on  memory.  Perhaps  the  best  documented 
phenomenon  is  mood-congruent  recall  in  memory  (Bower,  1981;  Blaney  1986);  that  is,  the 
observation  that  a  particular  affective  mood  induces  recall  of  similarly  valenced  memories  (e.g., 
depressed  mood  enhances  recall  of  negative  experiences  in  the  past,  including  negative  self¬ 
appraisals).  Depression  has  also  been  studied  in  the  context  of  particular  inferencing  tasks,  such 
as  judgment  and  decision-making.  In  these  tasks  depression  appears  to  lower  estimates  of  the 
degree  of  control  (Isen,  1993). 

Obsessiveness  A  number  of  studies  have  documented  the  impact  of  high-obsessiveness, 
characterized  by  "checking"  behavior,  on  cognitive  processing.  Among  the  primary  effects 
identified  are  the  following:  lack  of  confidence  in  own  attention  apparatus  to  capture  salient 
features  in  the  environment  (Broadbent  et  al.,  1986);  narrow  conceptual  categories  (Reed,  1969; 
Persons  and  Foa,  1984);  poor  memory  for  previous  actions  and  a  general  lack  of  certainty  about 
own  ability  to  distinguish  between  events  that  occurred  vs.  those  that  were  planned  or  imagined 
(Sher  et  al.,  1989),  and  slower  decision-making  speed  related  to  obsessive  gathering  of 
confirming  information  (Sher  et  al.,  1989). 

Emotional  stability  Emotional  stability  is  linked  to  a  predisposition  to  experience 
positive  or  negative  affect.  Low  emotional  stability  correlates  with  higher  rates  of  anxiety  and 
depression.  Studies  indicate  that  high  anxiety  leads  to  deficits  following  failure  while  low 


2The  impact  of  positive  states  has  also  been  studied  but  less  extensively.  For  example,  Isen  (1993)  has  found 
that  positive  affective  states  induce  greater  degree  of  elaboration  of  material  in  memory. 
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anxiety  leads  to  increased  effort  following  failure  (Revelle,  1990). 

Extraversion  Extraversion  correlates  with  positive  affect,  interacts  with  stress  to  affect 
performance,  and  is  associated  with  deficits  in  sustained  performance  (Revelle,  1997). 

Impulsivity  Impulsivity  appears  to  have  an  inverse  correlation  with  arousal;  less 
impulsive  individuals  have  a  higher  state  of  arousal.  This  facilitates  sustained  performance  over 
time,  but  apparently  hinders  performance  on  tasks  requiring  retention  of  high  amounts  of 
information  over  brief  time  periods.  Low  impulsives  show  lower  sensitivity  to  rewards  and  are 
slower  in  initiating  new  tasks  but  show  increased  persistence  once  the  task  is  begun  (Revelle, 
1987).  Results  of  empirical  studies  reveal  complex  patterns  of  interactions  indicating  effects  of 
impulsivity  on  performance  that  varies  with  stress  level,  task  demands,  and  time  of  day. 
Specifically,  impulsivity  seems  to  affect  both  short  and  long  term  memory,  persistence  in 
behavior  following  failure,  and  ability  to  maintain  sustained  information  transfer  (Revelle,  1997). 

The  research  summarized  above  provides  evidence  for  the  ubiquitous  impact  of  emotion  on 
cognitive  processing,  and  the  central  role  of  emotion  in  the  control  of  behavior.  The  emerging 
findings  also  begin  to  blur  the  distinction  between  what  has  traditionally  been  thought  of  as  the 
separate  realms  of  cognition  and  emotion. 

Table  2.2-1:  Impact  of  Emotion  and  Personality  Traits  on  Cognition: 

Summary  of  Empirical  Findings 

Anxiety  and  Attention 

Narrowing  of  attentional  focus 

Predisposing  towards  detection  of  threatening  stimuli 

Mood  and  Memory 

Mood-congruent  memory  phenomenon  -  particular  affective  state  induces  recall  of  similarly  valenced  material 

Obsessiveness  and  Performance 

Delayed  decision-making 

Reduced  ability  to  recall  recent  activities 

Reduced  confidence  in  ability  to  distinguish  among  actual  and  imagined  actions  and  events 
Narrow  conceptual  categories 

Affect  and  Judgment  &  Perception 

Depression  lowers  estimates  of  degree  of  control 

Anxiety  predisposes  towards  interpretation  of  ambiguous  stimuli  as  threatening 

Emotional  Stability 

Low  emotional  stability  correlates  with  higher  incidence  of  depression  and  anxiety 
Low  emotional  stability  correlates  with  decreased  effort  following  failure 

Extraversion 

Correlates  negatively  with  ability  to  sustain  performance 
Correlates  positively  with  positive  affect 

Impulsivity 

Correlates  negatively  with  arousal 
Low-impulsives  show  lower  sensitivity  to  rewards 

Low-impulsives  show  decreased  ability  to  initiate  tasks  but  increased  ability  to  sustain  performance  over  time 
Implications  for  ABAIS  Existing  evidence  summarized  above  provides  an  empirical 
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basis  for  predicting  the  generic  effects  of  emotional  states  and  personality  traits  on  performance. 
These  generic  effects  can  be  used  in  the  absence  of  specific  information,  and  also  serve  as 
guiding  principles  for  the  affective  /  cognitive  task  analysis  required  to  generate  specific 
performance  effects,  in  the  context  of  particular  situations. 

2.3  Individual  Differences  and  Personality  Research 

Two  additional  areas  of  psychological  research  relevant  to  assessment  of,  and  adaptation  to, 
affective  states  are  the  individual  differences  literature  and  the  personality  theory  literature. 
Each  of  these  areas  is  discussed  below. 

2.3.1  Individual  Differences  Research 

Recent  years  have  witnessed  a  renewed  interest  in  individual  differences  research  (Revelle, 
1995),  and  its  application  to  military  environments  (Pew  and  Mavor,  1998;  Deckert  et  al.,  1994; 
Illgen,  Zacharias,  Hudlicka  et  al.,  1997;  Fallesen,  1993).  For  example,  in  military  tactical 
environments,  empirical  studies  have  identified  a  number  of  differences  among  individuals  in 
terms  of  level  of  expertise,  cognitive  abilities  (specifically,  ability  to  perform  mental  'what-if 
simulations),  ability  to  visualize  situations  from  multiple  perspectives,  ability  to  abstract  relevant 
information,  and  preferred  style  of  information  presentation  (e.g.,  textual  vs.  visual,  abstract  vs. 
concrete,  etc.)  (Hammond,  Hamm,  Grassia  &  Pearson,  1987 ;  Deckert,  Entin,  Entin,  MacMillan 
&  Serfaty,  1994;  Badre,  1978).  Studies  of  tactical  planning  and  command  decision-making 
identify  features  that  characterize  expert  performance  and  thus  distinguish  between  expert  and 
novice  tactical  planners  (Lussier  et  al.,  1992;  Tolcott  et  al.,  1989;  Fallesen,  1993).  These  include 
factors  relating  to  situation  awareness  (awareness  of  uncertain  assumptions,  better  use  of 
available  information,  active  seeking  of  confirming  and  disconfirming  evidence,  greater 
awareness  of  enemy  activities,  and  awareness  of  more  critical  factors),  and  decision-making 
(more  flexible  planning,  more  elaborate  war-gaming,  explicit  prediction  of  events,  explicit 
consideration  of  adverse  alternatives  and  plan  failures).  A  number  of  additional  domain- 
independent  individual  differences  in  cognitive  styles  can  be  found  in  Hudlicka  (1997)  and  Illgen 
etal.  (1997). 

Recent  research  in  the  area  of  individual  differences  focused  on  the  investigation  of  the 
biological  basis  of  fundamental  dimensions  of  personality  (Revelle,  1995;  1997).  This  research 
focused  specifically  on  how  individual  differences  moderate  the  effects  of  situational 
manipulations  on  subsequent  cognitive  performance.  An  integrated  model  was  developed  of  how 
individual  differences  in  approach  and  avoidance  motivation  interact  with  situational 
manipulations  such  as  time  of  day,  time  on  task,  stimulant  drugs,  and  incentives,  to  affect 
working  memory,  subsequent  retrieval  from  long  term  memory,  and  sustained  performance 
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(Humphreys  and  Revelle,  1984).  Recent  efforts  focused  on  demonstration  of  how  stable 
individual  differences  in  diurnal  rhythms  can  be  measured  by  both  basal  body  temperature  and 
by  self  report  measures  of  energetic  and  tense  arousal,  and  how  these  latter  measures,  in  turn, 
predict  individual  patterns  of  changes  over  the  day  of  reaction  time  (Revelle  et  al.,  1997).  The 
multilevel  analysis  techniques  developed  can  be  applied  to  fit  performance  on  a  variety  of  tasks 
across  the  day  including  measures  of  basic  cognitive  performance  such  as  working  memory  and 
reaction  time  as  well  as  affective  judgments.  Individual  parameters  of  arousal  amplitude,  phase, 
and  coherence  have  been  shown  to  be  stable  across  several  weeks  and  to  show  predictable 
relationships  with  other  aspects  of  individual  differences. 

Implications  for  ABAIS  The  evidence  for  broad  variations  in  behavior  based  on 
individual  differences  underscores  the  importance  of  using  detailed  individual  histories  and 
baseline  performance  data  for  both  impact  prediction  and  strategy  selection.  The  specific  results 
help  identify  which  data  should  be  collected. 

2.3.2  Personality  Theory  Research 

Personality  research  has  had  a  long  and  controversial  history  in  academic  psychology.  The 
desire  to  predict  human  behavior  and  to  classify  the  complexities  of  human  nature  into  a  small 
number  of  categories  has  existed  in  folk  psychology  since  antiquity.  The  first  known 
classification  of  personalities  is  attributed  to  Hippocrates  around  400  BC,  who  classified 
individuals  into  four  temperaments:  choleric,  phlegmatic,  sanguine,  and  melancholic.  During  this 
centuiy,  personality  theories  and  personality  research  reflect  the  evolution  of  ideas  in 
psychological  theory  and  research  in  general,  specifically  the  controversy  between  behaviorism 
and  environmentally-determined  behavior  on  the  one  hand,  and  the  more  individually-focused 
theories  of  behavior  on  the  other.  Three  major  schools  of  psychological  thought  have  influenced 
personality  research  in  the  20th  century  and  gave  rise  to  the  major  theories  of  personality 
structure  and  development:  1)  Freud's  psychoanalytic  school  and  the  tri-partite  theory  of  the 
psyche  (id,  ego,  and  superego);  2)  behaviorist  theories  and  their  emphasis  on  learning  and 
environmentally-reinforced  behavior;  and  3)  humanistic  psychology  and  its  emphasis  on  the 
unique  enduring  characteristics  of  the  individual  and  the  interaction  between  individual  and 
environment  (McCrae  &  Costa,  1990). 

Of  these  theories,  the  latter  two  are  the  more  relevant  for  the  current  effort,  since  they 
provide  both  theoretical  detail  and  empirical  justification  for  application  to  the  analysis  human 
decision-making  behavior,  especially  in  military  contexts.  While  personality  research  has  been 
an  active  area  for  a  number  of  years,  with  a  number  of  journals  devoted  specifically  to  this  topic, 
it  should  be  kept  in  mind  that  many  of  the  most  fundamental  questions  asked  about  personality 
remain  open  research  issues  and  subjects  of  much  debate.  These  include  the  following  basic 
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questions: 

•  To  what  extent  is  behavior  predictable  across  situations? 

•  To  what  extent  can  personality  change  over  lifetime? 

•  What  are  the  basic  personality  traits? 

•  How  do  personality  traits  cluster  together  into  types? 

•  What  are  the  best  indicators  of  the  different  personality  types? 

•  How  do  personality  traits  interact  with  situational  variables? 

•  How  do  personality  traits  affect  cognition  and  decision-making? 

The  personality  research  field  has,  until  recently,  been  polarized  into  two  camps:  the  trait 
theorists ,  emphasizing  the  importance  of  stable,  persistent  personality  traits  across  situations,  and 
the  situationists  or  social  learning  theorists,  emphasizing  the  importance  of  situational  variables 
in  determining  individual  behavior.  Like  many  other  false  dichotomies,  the  trait-vs.-situation 
controversy  fueled  psychological  debate  over  several  decades  in  the  60's,  70's  and  early  80's. 
More  recently,  a  compromise  position  seems  to  have  been  reached,  termed  the  interactionist 
view  (Revelle,  1995;  Krahe,  1990).  We  will  summarize  current  personality  research  below  by 
presenting  the  extreme  positions  for  expository  purposes.  The  reader  should  keep  in  mind  that 
few  contemporary  researchers  adhere  to  either  extreme  point  of  view. 

2.3.2. 1  Trait  Theories 

The  pure  trait  theory  point  of  view  posits  the  existence  of  fixed  personality  predispositions 
for  behavior,  the  personality  traits,  which  are  stable  across  time  and  across  situations.  The 
number,  and  to  some  extent  the  nature,  of  the  basic  traits  vary,  generally  between  three  and  seven 
but  in  some  cases  going  up  to  sixteen  distinct  traits.  Table  2.3.2. 1-1  includes  examples  of  traits 
identified  by  various  psychologists. 


Table  2.3.2.1-1:  Personality  Traits 


Hippocrates  (400 
BC) 

Choleric 

Melancholic 

Phlegmatic 

Sanguine 

Jung  (20’s-40’s) 
Extroverted  vs.  Introverted 
Sensing  vs.  Intuiting 
Thinking  vs.  Feeling 

Judging  vs.  Perceiving 

Eysenck  <60’s-80’s) 

Neuroticism 

Extroversion  vs.  Introversion 
Psychoticism 

Costa  and  McCrae 
(80’s-90’s) 

Neuroticism  vs.  Emotional 
Stability 

Extroversion  vs.  Introversion 
Openness 

Conscientiousness 

Agreeableness 

The  traits  are  generally  identified  by  a  factor  analysis  of  self-report  data  collected  from  subjects 
who  fill  out  a  variety  of  personality  inventory  instruments.  Subjects  are  asked  to  answer  a 
number  of  questions  about  how  they  have  or  might  react  in  certain  situations,  how  they  feel  and 
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what  they  think  about  different  aspects  of  themselves,  others,  and  their  lives.  Other  data  are  also 
used  as  sources,  including  observations  of  behavior  in  naturalistic  life  situations  and  observations 
of  behavior  in  situations  designed  for  experimental  purposes. 

To  the  extent  that  the  personality  traits  can  be  identified,  behavior  of  an  individual  can  be 
predicted  in  a  given  situation.  Not  only  has  the  number  and  type  of  the  traits  varied  over  the 
years,  but  different  researchers  have  constructed  various  categorizations,  depending  on  the  focus 
of  their  research  and  their  findings.  For  example,  Allport  and  Cattell  distinguish  traits  by 
frequency  of  occurrence  and  identify  common  and  unique  traits.  Cattell  also  distinguishes 
between  source  traits,  which  are  more  fundamentally  causal,  and  surface  traits,  which  are  the 
manifested  effects  of  the  source  traits  (Mischel,  1976).  Examples  of  source  traits  are  dominance 
vs.  submissiveness,  and  ego  strength  vs.  emotionality  and  neuroticism.  Examples  of  surface 
traits  are  integrity  and  altruism  vs.  dishonesty  and  undependability,  and  thoughtfulness  vs. 
foolishness.  Cattell  also  categorizes  traits  by  their  contribution  to  behavior.  Thus  he  identifies 
dynamic  traits  as  those  relevant  in  initiating  action  and  behavior,  ability  traits  describe  specific 
skills  used  in  satisfying  the  individual's  goals,  and  temperament  traits  are  traits  related  to 
emotional  reactivity.  Cattell  also  categorized  traits  as  to  their  source  as  environmental-mold  vs. 
constitutional,  and  their  stability  across  situations  (general  vs.  specific).  In  this  latter 
categorization  he  foresaw  the  eventual  merging  of  the  trait  theory  and  situational-social  learning 
theory  points  of  view. 

Recent  trait  theory  research,  exemplified  by  the  work  of  Costa  and  McCrae  (1989),  shows 
evidence  for  five  traits  that  remain  stable  in  individuals  across  the  lifetime.  This  five  factor 
personality  model,  known  as  the  "Big  5",  and  its  associated  instrument,  the  NEO-PI,  has  become 
the  most  widely  accepted  trait  theory  model  with  significant  supporting  empirical  evidence.  The 
NEO-PI  also  seems  to  adequately  account  for  traits  identified  by  other  models  and  inventories 
(McCrae&  Costa,  1989;  McCrae  &  Costa,  1990). 

A  more  biologically-oriented  model  has  been  suggested  by  Eysenck  (1967;  1991)  and  posits 
three  dimensions:  extraversion,  neuroticism,  psychoticism,  with  psychoticism  being  related  to 
aggressiveness.  This  model  is  often  referred  to  as  the  “Giant  3”. 

While  academic  psychologists  have  struggled  with  fundamental  questions  such  as  the 
number  and  nature  of  the  basic  personality  traits,  and  to  what  extent  personality  can  change  over 
lifetime,  on  the  more  applied  side  management  consultants,  career  counselors,  and  human 
resource  professionals  have  focused  on  the  application  of  one  model  and  its  associated 
instrument:  the  Jungian-psychology  based  Myers  Briggs  Type  Indicator.  This  instrument  has 
been  applied  primarily  in  work  settings  for  team  management,  team  composition,  team  training, 
and  career  counseling.  The  MBTI  is  based  on  the  Jungian  dimensions  of  personality  and  Jung's 
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conception  that  opposing  tendencies  exist  in  the  personality  which  are  eventually  integrated  in 
the  well-functioning  individual.  Jung  identified  several  categories  of  dimensions,  the  four  used 
most  commonly  are:  1)  introversion  vs.  extraversion  -  describing  individual's  orientation  towards 
others;  2)  thinking  vs.  feeling  -  describing  individual's  orientation  toward  emotions;  3)  sensing 
vs.  intuition  -  describing  individual's  orientation  towards  stimuli  and  information  processing;  and 
4)  judging  vs.  perceiving  -  describing  the  individual's  attitude  toward  structure.  The  MBTI  has 
not  been  as  widely  investigated  in  academic  circles,  possibly  due  to  its  origins  by  a  non¬ 
psychologist  mother-daughter  team.  There  is  however  an  organization  promoting  its  use,  the 
Association  for  Psychological  Types,  which  holds  annual  conferences  and  publishes  a  quarterly 
research  journal:  The  Association  for  Psychological  Types  Research  Journal.  In  spite  of  its 
marginal  status  in  psychological  research,  several  studies  of  the  MBTI  have  been  conducted 
(Druckman  &  Bjork,  1991)  and  a  number  of  researchers  have  made  comparisons  of  the  MBTI  to 
the  Big  5  and  other  factor  models,  indicating  close  correspondences  between  the  two  (McCrae 
and  Costa,  1990). 

2.3. 2.2  Situationist  Models  and  Social  Learning  Theories 

In  part  as  a  natural  consequence  of  the  behaviorist  tradition  based  on  learning  theories,  and  in 
part  as  a  reaction  to  the  often  contradictory  data  evident  in  the  trait  models,  situationist  and 
social  learning  theories  suggest  an  alternative  model.  They  propose  that  the  individual’s  behavior 
is  determined  by  situational  variables  and  by  the  individual's  experience  in  similar  situations  in 
the  past  (reinforcement),  rather  than  by  a  stable  predisposition  to  act  in  a  certain  way  across 
situations.  Thus  the  observations  of  consistent  behavior  across  situations,  observations  which 
could  lead  to  the  inference  of  stable  traits,  can  in  fact  be  interpreted  as  the  result  of  learned  and 
previously  reinforced  behavior  in  similar  situations  (Mischel,  1976).  Thus  an  aggressive 
behavior  need  not  be  interpreted  as  a  result  of  a  hostile  personality  but  rather  as  the  result  of 
previously  reinforced  aggressive  behavior.  The  initial  radical  behaviorist  position  that  situational 
variables  and  reinforcement  history  were  solely  responsible  for  behavior,  was  later  modified  to 
include  individual  variables.  Individual  variables  include  specific  abilities  and  competencies  (IQ, 
skills,  etc.),  encoding  strategies  (selecting  attention,  interpretation  and  categorization),  internal 
mental  representations,  expectations  about  outcomes  resulting  from  previous  learning,  subjective 
values,  and  current  goals  and  plans.  Perhaps  the  major  contribution  of  social  learning  theories  is 
that  the  best  predictor  of  future  behavior  is  past  behavior. 

Much  as  the  trait  theorists'  acknowledgment  that  situational  variables  play  a  role  in  behavior, 
in  addition  to  the  individual's  predispositions  to  behave,  think  and  feel  in  a  particular  way,  the 
social  learning  theorists'  acknowledgment  that  individual  variables  play  a  role  in  behavior  is  a 
step  towards  the  eventual  reconciliation  of  the  two  extreme  views.  This  view,  the  interactionist 
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view  (Krahe,  1990),  represents  a  modem  synthesis  of  the  earlier  extreme  views  and  promises  to 
enrich  personality  research  by  its  broader  focus  that  includes  both  the  individual  and  the 
situation,  and  the  interaction  between  the  behavior  triggering  stimuli  and  the  individual 
personality  traits. 

23.2.3  Fighter  Pilot  Personality  Assessment 

As  is  the  case  in  many  fields,  personality  assessment  in  the  Air  Force  “has  a  long  and 
controversial  history”  (Callister  et  al.,  1997,  p.  1).  Personality  measures  show  mixed  ability  to 
predict  performance.  On  the  one  hand,  they  seem  to  fail  to  predict  completion  of  initial  training, 
on  the  other  hand,  they  do  appear  to  predict  ratings  of  pilots  in  commercial  settings  and  retention 
of  USAF  pilots  (Callister  et  al.,  1997).  The  identification  of  an  ‘ideal  pilot’  personality  is 
similarly  controversial.  For  example,  early  World  War  I  studies  suggest  that  both  a  “high- 
spirited  and  happy-go-lucky”  and  “quiet  and  methodical”  represent  ideal  pilot  profiles  (Callister 
et  al.,  1997,  p.  1).  It  is  interesting  to  note  that  as  the  military  environment  changes  from 
individual-based  to  increasingly  team-based,  the  “ideal”  personality  types  for  a  variety  of  tasks 
may  change  as  well,  from  the  more  independent,  individually-oriented  type  to  a  more  socially- 
oriented  type  of  personality. 

A  number  of  recent  studies  exist  assessing  the  pilot  population  in  terms  of  a  variety  of 
standard  psychological  assessment  instruments  (e.g.,  NEO-PI-R,  MMPI,  etc.)  (Callister  et  al., 
1997),  as  well  as  instruments  specialized  for  the  fighter  pilot  population  (e.g.,  ALAPS  (Retzlaff 
et  al.,  1997)).  In  one  study,  Callister  and  colleagues  (1997)  tested  1301  USAF  student  pilots 
(92%  male)  using  the  NEO-PI-R  inventory  based  on  the  Big  5  personality  factors  (neuroticism, 
extraversion,  openness,  agreeableness,  and  conscientiousness).  The  scores  were  compared  with 
those  of  general  population.  The  results  indicate  that  both  males  and  females  differ  on  the 
extraversion  (higher  than  general  population)  and  agreeableness  dimension  (lower  than  general 
population),  as  well  as  in  assertiveness,  and  excitement-seeking  (higher  than  general  population). 
In  addition,  the  female  USAF  component  of  this  sample  shows  a  higher  score  on  openness  than 
the  general  population. 

While  studies  using  standard  psychological  instruments  are  useful,  they  often  fail  to  focus  on 
the  specific  traits  relevant  to  specific  USAF  tasks,  or  do  not  meet  the  desirable  psychometric 
norms.  To  help  address  these  shortcomings,  a  specialized  instrument  has  been  developed  at 
Armstrong  Laboratory  to  assess  the  aviation  population.  This  instrument,  the  Armstrong 
Laboratory  Aviation  Personality  Survey  (ALAPS),  consists  of  240  true  or  false  items  divided 
into  15  subscales.  ALAPS  is  described  in  greater  detail  in  section  2.4  below. 

Implications  for  ABAIS  Personality  research  provides  a  number  of  relevant 
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contributions  for  ABAIS.  First,  it  confirms  to  a  large  extent  the  folk  psychology  notion  that  the 
best  predictor  of  future  behavior  is  past  behavior.  This  underscores  the  importance  of  gathering 
as  much  information  as  possible  about  the  ABAIS  user's  past  performance  and  using  this 
information  as  part  of  the  individual  history  category  of  factors  to  assess  affect  and  belief  states, 
and  to  predict  likely  and  performance  biases.  Second,  it  helps  identify  a  set  of  personality  traits 
to  use  as  part  of  the  ABAIS  user  profile  description.  The  traits,  while  not  strictly  orthogonal,  do 
provide  a  good  coverage  for  a  broad  range  of  personalities  and  help  predict  generic  classes  of 
behaviors.  The  traits  selected  for  the  Phase  I  ABAIS  user  profile  were  derived  from  a 
combination  of  the  Big  5  and  the  "Giant  3",  and  filtered  by  their  potential  relevance  to  the  USAF 
tasks  in  general,  and  the  Phase  I  demonstration  task  in  particular.  The  selected  traits  are: 
emotional  stability,  impulsiveness,  risk  tolerance,  aggressiveness,  and  obsessiveness.  Third,  the 
findings  from  interactionist  theories  suggest  the  importance  of  including  in  the  individual  history 
specific  trait-situation  interactions  (e.g.,  anxiety  only  occurs  during  defensive  maneuvers  but  not 
during  offensive  maneuvers  for  a  particular  pilot)  and  use  this  type  to  predict  pilot  behavior  in 
similar  situations.  Fourth,  these  findings  help  identify  specific  expressions  of  a  particular  trait 
(e.g.,  anxiety  causes  physical  symptoms  but  has  no  impact  on  cognitive  abilities)  and  use  this 
information  to  predict  the  impact  of  particular  affective  states  on  the  pilot's  decision-making. 
Fifth,  personality  theory  offers  a  series  of  standardized  clinical  instruments  to  identify  specific 
affective  states  and  temperamental  traits,  as  discussed  in  section  2.4.3  below. 

2.4  Assessment  of  Affective  States 

2.4.1  Self  reports  and  Psychological  Instruments 

While  a  seemingly  countless  number  of  personality  inventories  exist,  they  can  be  divided  into 
two  broad  categories:  those  targeted  at  measuring  personality  structure  in  terms  of  the  small 
number  of  factors  identified  by  the  various  trait  theories,  and  those  focusing  on  psychopathology 
in  general  or  on  a  particular  set  of  symptoms.  We  discuss  both  categories  below. 

Personality  Assessment  Personality  inventories  exist  for  each  of  the  factor  models 
mentioned  discussed  in  section  2.3.2.  However,  the  two  inventories  used  most  frequently  are  the 
NEO-PI  (Costa  and  McCrae,  1989),  based  on  Costa  and  McCrae's  Big  5  model,  and  the  MBTI 
(Briggs  and  Myers,  1977),  based  on  Jung's  model  of  personality.  The  NEO-PI-R  is  more  broadly 
accepted  in  the  academic  community  and  has  been  subjected  to  extensive  validation  studies. 
NEO-PI-R  consists  of  240  items  evaluated  on  a  5-point  Likert  scale  (“strongly  agree”,  “agree”, 
“neutral”,  etc.).  The  test  provides  scores  for  each  of  the  Big  5  factors  -  neuroticism,  extraversion, 
openness,  agreeablenes,  and  conscientiousness.  Each  of  these  major  scores  consists  of  six 
subscales  (see  table  2.4.1-1). 
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Table  2.4.1-1:  NEO-PI-R  Personality  Assessment  Scales  and  Subscales 


Neuroticism 

Extraversion 

Openness 

Agreeableness 

Conscientiousne 

Anxiety 

Warmth 

Fantasy 

Trust 

ss 

Angry  Hostility 

Gregariousness 

Aesthetics 

Straighforwardness 

Competence 

Depression 

Assertiveness 

Feelings 

Altruism 

Order 

Self-Consciousness 

Activity 

Actions 

Compliance 

Dutifulness 

Impulsiveness 

Excitement-Seeking 

Ideas 

Modesty 

Achievement  Striving 

Vulnerability 

PositiveEmotioanility 

Values 

Tenderness 

Self-Discipline 

Deliberation 

The  MBTI  is  generally  used  in  more  applied  settings  for  management  purposes,  career 
counseling,  and  team  training,  as  well  as  in  popular  psychology  (e.g.,  matchmaking).  The  MBTI 
consists  of  126  items  and  the  result  consists  of  eight  scores,  corresponding  to  each  of  the  eight 
poles  of  the  Jungian  personality  model:  Introverted-Extraverted;  Sensing-Intuitive;  Thinking- 
Feeling;  and  Judging-Perceiving. 

Of  particular  interest  is  the  aviation-oriented  instrument  developed  by  USAF  psychologists, 
the  Armstrong  Laboratory  Aviation  Personality  Survey  (ALAPS).  ALAPS  consists  of  240  true 
or  false  items  divided  into  15  subscales.  The  subscales  cover  personality,  psychopathology,  and 
crew  interaction  (Retzlaff  et  al.,  1997).  Summary  of  the  instrument  is  shown  in  table  2.4. 1-2. 
This  instrument  has  been  successfully  validated  and  is  currently  being  used  in  on-going 
personality  assessment  research. 


Table  2. 4.1-2:  Aviation-Oriented  ALAPS  Personality  Assessment  Instrument 


Confidence  /  Narcissism 

Affective  Lability 

Deference  /  Submissiveness 

Socialness 

Anxiety 

Team  Orientation 

Aggressiveness 

Depression 

Organization 

Orderliness  /  Compulsivity 

Alcohol  Abuse 

Impulsivity 

Negativity 

Dogmatism  /  Authoritarianism 

Risk  Taking 

Psychopathology  Assessment  Probably  the  most  widely  used  instrument  in  clinical 
setting  is  the  Minnesota  Multiphasic  Personality  Inventory  (MMPI)  (Hathaway  and  McKinley, 
1989),  which  consists  of  567  items  and  classifies  individuals  into  ten  scales:  hysteria,  depression, 
psychopathic  deviate,  masculinity-femininity,  paranoia,  psychasthenia,  schizophrenia,  social 
introversion.  The  MMPI  also  includes  three  control  scales  to  detect  consistent  biases  in  the  data: 
lie  scale,  defensiveness  scale,  careless/confused. 

In  addition  to  the  broadly-oriented  MMPI,  there  is  a  vast  and  ever  increasing  number  of 
questionnaires  focusing  on  specific  affective  states.  These  include  anxiety  (The  State-Trait 
Anxiety  Scale,  The  Manifest  Anxiety  Scale),  panic  attack  and  phobia  sensitivity  (Anxiety 
Sensitivity  Index  (Peterson  &  Reiss,  1987)),  negative  emotionality  (Positive  and  Negative  Affect 
Scales  —  PANAS  (Watson,  Clark,  Tellegen,  1988)),  depression  (Beck  Depression  Inventory), 
stress  coping  strategies,  locus  of  control,  eating  disorders  and  a  number  of  others. 
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Implications  for  ABAIS  Both  personality  theory  and  psychopathology  offer  a  series  of 
standardized  and  validated  clinical  instruments  to  identify  specific  affective  states  and 
temperamental  traits  from  self-report  data.  While  the  pathologically-oriented  instruments  are 
less  relevant  for  the  USAF  population,  particularly  the  fighter  pilots,  the  personality  instruments 
in  general  do  provide  useful  data  about  specific  relevant  traits  and  potential  susceptibility  to 
specific  affective  states.  These  instruments  can  be  used  in  two  ways  as  part  of  the  self- 
assessment  procedure  in  ABAIS.  First,  during  an  off-line,  initial  assessment,  to  provide 
background  information  and  suggest  generic  effects  on  performance.  Second,  during  brief  on¬ 
line  assessment,  where  one  or  two  items  would  be  presented  to  the  user  during  task  performance, 
to  provide  a  specific,  targeted  assessment  of  their  current  affective  state.  This  latter  application 
of  self-reports  would  need  to  be  compatible  with  the  task  context  and  would  not  be  appropriate  in 
all  situations. 

2.4.2  Physiological  Methods 

The  fundamental  questions  regarding  physiological  assessment  of  affective  states  is  this:  Do 
different  emotions  exhibit  unique  physiological  manifestations?  From  a  historical  perspective, 
the  answer  to  this  question  evolved  over  the  past  100  years  (Ekman  and  Davidson,  1994). 
According  to  William  James  (1890),  the  answer  was  a  'yes':  “distinct  emotions  associated  with 
unique  patterns  of  skeletal  muscle  and  physiological  changes  since  our  experience  of  the  emotion 
is  a  direct  function  of  feedback  from  the  periphery”  (Davidson,  1994,  p.  237).  In  the 
predominantly  cognitively-oriented  60's  and  70's,  Schachter  and  Singer's  (1962)  answer  was  'no' 
and  their  two-factor  theory  of  emotion  suggested  that  the  same  state  of  undifferentiated  arousal 
can  lead  to  different  emotions,  depending  on  the  activated  cognitions.  Contemporary  researchers 
vary  in  their  response  to  this  question  and  the  answer  appears  to  be  a  qualified  'yes,  sometimes' 
and  'yes,  if  you  look  in  the  right  place  (i.e.,  the  brain  not  the  peripheral  nervous  system)  and 
using  the  correct  tool  (e.g,  electrodes,  fMRI,  PET  scans,  etc.).  "To  the  extent  emotions  are 
associated  with  different  action  tendencies,  they  should  differ  in  their  autonomic  patterns” 
(Davidson,  1994,  p.  241).  This  summary  will  be  elaborated  in  more  detail  below. 

The  vast  majority  of  efforts  to  identify  unique  physiological  signatures  for  specific  emotions 
have  focused  on  the  autonomic  nervous  system  and  facial  muscles,  although  some  work  has  also 
been  done  using  data  from  the  endocrine  system,  EEG,  and,  more  recently  brain  scan  data  are 
being  studied,  such  as  fMRI  and  PET  scans.  The  most  frequently  studied  ANS  signals  include: 
heart  rate,  heart  rate  variability,  respiration  rate,  EEG,  skin  conductance  (GSR),  pupillary 
dilation,  and  blood  volume  pressure  (BVP).  Facial  EMG  is  used  to  detect  muscle  movement  and 
eye  blinks.  While  a  number  of  these  measures  successfully  assess  arousal  (heart  rate,  skin 
conductance,  pupillary  dilation)  and  valence  (facial  EMG  -  corrugator  &  zygomatic  muscles). 
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the  results  in  further  differentiation  among  the  different  affective  states  are  mixed. 

When  considering  the  2-dimensional  model  of  emotion  discussed  above,  that  is,  arousal  and 
valence,  a  variety  of  signals  appear  to  reliably  reflect  high  and  low  states  of  arousal,  and  positive 
and  negative  valence.  Specifically,  heart  rate,  skin  conductance  measures,  and  pupillary  dilation 
correlate  with  arousal;  activity  of  the  corrugator  and  zygomatic  muscle  groups  correlates  with 
valence.  Although  these  dimensions  can  be  successfully  assessed  in  principle,  many  practical 
issues  exist  in  applying  the  physiological  affect  assessment  methods  in  practice.  These  include: 
intrusiveness  of  the  measurement  apparatus  (e.g.,  skin  conductance  and  facial  EMG  sensors), 
restrictions  of  the  subject’s  movement  (e.g.,  EEG  sensors),  task  environment  which  may  interfere 
with  the  measurement  (e.g.,  perspiration  interfering  with  collection  of  skin  conductance  data), 
difficulty  gathering  the  required  data  in  the  task  context  (e.g.,  pupil  dilation  detection  in  a  fighter 
pilot  cockpit  (Callan,  1998)),  individual  differences  in  physiological  reactivity,  and  day-to-day 
and  diurnal  variations  in  physiological  activity  in  general.  This  type  of  variability  further 
underscores  the  importance  of  adequate  baseline  data  for  each  individual,  to  accurately  map  the 
subject’s  affective  signals  onto  a  specific  state. 

The  picture  is  not  as  clear  when  considering  the  more  complex  space  of  emotions,  such  as  the 
basic  emotions.  While  some  researchers  report  results  indicating  the  ability  to  differentiate 
between  positive  and  negative  emotions  (Levenson  et  al.,  1990),  others  have  been  unable  to 
duplicate  these  efforts  (Ekman  and  Davidson,  1994,  p.  261-262).  Levenson  and  colleagues 
(1990)  report  data  that  differentiate  between  the  following  emotions:  disgust,  fear,  anger, 
sadness,  happiness.  The  theoretical  explanations  of  these  data  suggest  that  these  findings  are  due 
to  different  metabolic  requirements  for  the  different  types  of  behaviors  associated  with  these 
emotions.  Thus  fear  and  anger  require  higher  metabolic  activity  than  disgust  and  sadness.  Anger 
and  fear  produce  larger  heart  rate  acceleration  than  happiness,  presumably  because  they  must 
prepare  the  organism  for  flight-or-flight  in  the  former  case,  but  not  in  the  latter.  Positive 
emotions  have  been  hypothesized  by  Levenson  to  function  as  “undoers”  of  the  autonomic 
arousal  and  metabolic  behavioral  preparations  caused  by  the  more  negative  emotions;  that  is, 
positive  emotions  function  to  rapidly  return  organism  to  its  pre-arousal  state. 

Several  researchers  suggest  that  past  failures  to  identify  unique  physiological  correlates  may 
be  a  function  of  having  "looked  in  the  wrong  places"  (Gray,  1994).  Future  work  will  focus  on 
the  brain  and  use  new  technologies  (neuroimaging,  single-cell  &  cell  cluster  recordings, 
neuropeptide  analysis),  taking  advantage  of  recent  findings  in  neuroscience.  For  example,  left 
anterior  prefrontal  cortex  appears  to  be  associated  with  positive-affect  and  approach  behaviors, 
while  the  right  prefrontal  cortex  is  associated  with  negative-affect  and  withdrawal  (Davidson, 
1992);  the  amygdala  neurons  and  neuronal  clusters  appear  to  differentiate  between  rewarding 
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and  non-rewarding  situations  (LeDoux,  1994);  different  neuropeptides  associated  with  different 
emotions  (Panksepp,  1994)  such  as  anxiety,  aggressiveness,  social  processes,  playfulness, 
fearfulness.  While  these  findings  promise  to  provide  much  more  detailed  reflections  of  affective 
states,  many  technical  problems  must  be  overcome  before  recordings  and  signal  analysis  from 
these  areas  can  be  applied  in  real-time  settings  for  the  detection  of,  and  adaptation  to,  a  variety  of 
affective  states. 

Implications  for  ABAIS  While  a  number  of  experimental  methods  are  theoretically 
available,  results  of  the  literature  search  reveal  that  most  reliably  assessed  affective  measures  are 
arousal  and  valence.  The  best  practical  signal  for  arousal  detection  is  heart  rate  (Hugdahl,  1995; 
Orr,  1998;  Hofmann,  1998;  Cacioppo  et  al.,  1993).  Other  measures  of  arousal,  such  as  galvanic 
skin  response,  pupil  size,  blood  volume  pressure,  etc.,  either  do  not  provide  additional  data  and  / 
or  are  not  as  readily  assessed.  While  skin  conductance  measures  represent  a  better  direct 
measure  of  anxiety,  the  requirement  of  finger  or  palm  sensors  makes  these  impractical  in 
computerized,  automated  environments.  Heart  rate  variability  is  a  highly  useful  measure,  but  is 
more  concerned  with  tonic  arousal  rather  than  the  temporary,  phasic  arousal  due  to  transient 
anxiety  states,  which  are  of  interest  in  the  proposed  task  context  (Cacioppo  et  al.,  1993;  Orr, 
1998;  Hofmann,  1998).  The  best  means  of  assessing  valence  is  facial  EMG,  using  the  corrugator 
and  zygomatic  muscle  groups.  This  approach  will  be  discussed  in  greater  detail  in  section  2.4.3 
below. 

2.4.3  Facial  Recognition 

Successful  use  of  facial  recognition  as  an  affect  assessment  method  is  predicated  on  two 
assumptions.  First,  that  distinct  emotions  produce  distinct  configurations  of  the  facial 
musculature.  Second,  that  these  configurations  can  be  recognized,  either  through  the  use  of 
electromyograms  or  using  computer  vision  pattern  recognition  methods.  Both  of  these  issues  are 
discussed  below. 

Distinct  Facial  Expressions  as  Emotion  Markers  The  area  of  facial  expression  of  emotion 
has  been  studied  extensively,  beginning  with  Tomkins’  work  and  continuing  with  contemporary 
work  of  Fkman  and  colleagues  and  Izard  (Cacioppo  et  al.,  1993).  In  fact,  much  of  the  evidence 
for  the  existence  of  basic  emotions  comes  from  facial  expression  research.  Strong  empirical 
evidence  exists  supporting  the  existence  of  cross-cultural  regularities  in  the  expression  of  a 
number  of  emotions. 

The  primary  muscle  groups  of  interest  in  emotion  expression  are  the  corrugator  (“eyebrow 
muscle”)  and  the  zygomatic  (“smile  muscle”).  Negative  emotions  such  as  fear,  sadness,  and 
anger  are  reflected  in  higher  corrugator  muscle  activity.  Positive  emotion  (i.e.,  happiness)  is 
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reflected  in  higher  zygomatic  muscle  activity.  While  these  methods  appear  to  be  robust,  there  are 
issues  about  data  interpretation.  For  example,  some  problems  are  associated  with  interpreting  the 
zygomatic  movement  however,  since  some  negative  emotions  also  cause  movement  in  that 
muscle  group.  It  is  not  clear  whether  this  is  due  to  “distress  smiling”  or  cross-talk  between 
muscle  groups  (Cacioppo  et  al.,  1993). 

A  number  of  coding  schemes  exist  for  classifying  emotions  based  on  facial  expression,  the 
Facial  Action  Coding  System  (FACS)  developed  by  Ekman  and  colleagues  being  the  most 
comprehensive  (Ekman  and  Friesen,  1975). 

Recognition  of  Facial  Expressions  Two  methods  used  for  facial  expression  recognition  are 
facial  EMG  and  computer  vision  pattern  recognition.  Facial  EMG  involves  the  attachment  of 
either  electrodes  or  some  other  sensor  (e.g.,  piezoelectric  tape)  to  key  locations  of  the  face,  so 
that  minute  muscle  movements  can  be  identified  as  they  cause  changes  in  the  facial  skin 
configuration.  A  recent  example  of  the  use  of  piezoelectric  tape  is  work  by  Riseberg  and  Picard 
(1999),  where  the  corrugator  muscle  movement  was  used  to  differentiate  between  interest  and 
surprise  (“upward”  movement  of  the  muscle),  and  confusion  and  frowning  (“downward” 
movement).  While  the  work  is  still  in  progress,  initial  data  show  promise  in  the  ability  to 
discriminate  between  these  emotion  categories,  using  a  relatively  non-intrusive  assessment 
method. 

Computer  vision  pattern  recognition  uses  a  camera  to  identify  distinct  emotions  from  either 
still  images  or  short  video  segments.  Generally,  video  segments  are  preferred  over  still  images 
for  facial  recognition,  as  more  temporal  information  is  provided  which  helps  disambiguate  the 
data  (Picard,  1997).  Several  recent  efforts  have  successfully  discriminated  among  different 
affective  states,  with  a  high  degree  of  accuracy.  For  example,  Essa  and  Pentland  (1997),  and 
Yacoob  and  Davis  (1996),  report  discriminations  between  the  basic  emotions  with  up  to  98% 
accuracy.  However,  several  factors  limit  the  applicability  of  these  results  to  a  real-time  affective 
adaptive  interface.  First,  the  number  of  subjects  in  both  cases  was  small  (8-12),  bringing  into 
question  the  robustness  of  the  results;  second,  the  emotions  that  were  recognized  were  artificial, 
in  the  sense  that  the  individuals  were  asked  to  display  a  particular  emotion  in  one  case,  and  faces 
of  TV  actors  were  used  in  the  other  case;  third,  the  time  required  for  recognition  was  too  long  for 
real-time  assessment  (up  to  5  minutes  per  face). 

Implications  for  ABAIS  While  pattern  recognition  approach  is  theoretically  possible 
and  promising  work  exists  in  the  area,  this  approach  is  both  computationally  intensive,  and  not 
sufficiently  robust  at  this  stage  for  real-time  affective  assessment.  More  promising  is  the 
assessment  of  valence  via  facial  EMG,  specifically,  via  the  movement  of  the  corrugator  and 
zygomatic  muscle  groups.  These  approaches  are  promising,  especially  given  the  emerging  non- 


34 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


intrusive  technologies  such  as  piezoelectric  tapes.  While  it  is  difficult  to  reliably  differentiate 
among  a  large  set  of  emotions,  the  physiological  assessment  valence  appears  via  facial  EMG. 

2.5  Situation  Awareness 

Much  recent  research  in  decision  making  and  skilled  human  performance,  particularly  in 
dynamic,  real-time  settings,  has  focused  on  the  concept  of  situation  assessment  and  situation 
awareness  (Endsley,  1995).  Briefly,  situation  awareness  refers  to  the  individual’s  ability  to 
rapidly  identify  salient  cues  in  the  incoming  data  and  map  those  cues  onto  a  small  set  of  relevant 
situations,  which  then  guide  further  action  selection.  A  series  of  extensive  studies  of  situation 
assessment  in  the  military  and  other  real-time  settings  have  been  conducted  by  Klein  and 
colleagues  (1989).  Klein  has  labeled  this  process  recognition-primed  decision-making  (RPD), 
and  identified  RPD  as  a  key  element  in  tactical  planning. 

To  the  extent  that  affective  state  and  personality  traits  influence  attention,  perception,  and 
cognition,  they  play  a  major  role  in  influencing  all  aspects  of  recognition-primed  decision¬ 
making  and  situation  assessment;  from  cue  identification  and  extraction,  to  situation 
classification,  and  finally  decision-selection.  In  the  battlefield  management  and  tactical  planning 
domain,  several  studies  of  human  performance  have  identified  a  number  of  types  decision  errors 
and  biases  in  tactical  situation  assessment  (Tolcott,  Marvin  &  Lehner,  1989;  Fallesen,  1993) 
which  contribute  to  the  inadequate  development  of  tactical  alternatives,  or  to  the  selection  of  an 
inappropriate  final  course-of-action  (COA). 

Empirical  studies  of  tactical  planning  and  decision-making  indicate  that  certain  categories  of 
failures  are  common,  resulting  in  inadequate  COA  development  and  selection  (Fallesen,  1993). 
Fallesen  divides  these  failures  into  categories  according  to  the  stage  of  the  COA  development 
process  (e.g.,  situation  assessment,  formulation  of  alternative  COAs,  comparison  of  these 
alternatives,  wargaming,  etc.).  For  each  category  he  then  identifies  the  most  critical  factors  that 
contribute  to  ineffective  and  non-optimal  performance.  Examples  of  these  factors  are  failures  to 
use  systematic  comparison  strategies  for  alternative  COAs,  failures  to  verify  uncertain 
information,  failures  to  develop  adequate  action/reaction  trees  due  to  inadequate  wargaming, 
failures  to  consider  all  factors,  failures  to  verify  assumptions,  failures  to  assess  information 
quality,  failures  to  interpret  available  information,  and  failures  to  make  predictions  for  situation 
assessment.  Other  research  indicates  that  knowledge  of  enemy  activities  is  particularly  critical 
and  often  neglected  by  tactical  planners. 

Of  particular  importance  is  the  primacy  bias,  that  is,  selecting  an  a  priori  option  and  then 
looking  for  confirmatory  evidence  and  ignoring  disconfirming  evidence  for  that  option.  Another 
common  bias  is  success  orientation,  that  is,  the  overconfidence  in  friendly  plans  and 
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underestimation  of  possible  enemy  activities  that  could  jeopardize  projected  friendly  activities 
(Fallesen  &  Michel,  1991;  Lussier,  Solick  &  Keene,  1992). 

The  general  area  of  cognitive  biases  has  been  studied  extensively  in  laboratory  settings,  and 
is  particularly  relevant  to  situation  assessment.  Cognitive  biases  result  from  use  of  cognitive 
heuristics,  many  of  which  may  be  unconscious  (Tversky  &  Kahneman,  1974).  The  following 
types  of  cognitive  biases  have  been  identified: 

•  Availability  -  Items  (e.g.,  events,  entities,  relations)  most  easily  retrieved  from  memory  are 

considered  most  important 

•  Order  effects  -  Recency/Primacy  bias  -  Initial  or  most  recent  information  is  considered  most 

important 

•  Representativeness  -  Likelihood  of  events  is  judged  by  their  similarity  to  typical  events, 

rather  than  by  statistical  frequency 

•  Internal  coherence  -  Information  consistent  with  previous  experience  and  beliefs  is  preferred 

•  Consistency  -  Preference  for  consistent  information  and  exclusion  of  important  evidence  to 

the  contrary 

•  Confirmation  bias  (cognitive  hysteresis,  set,  functional  fixation)  -  Search  for  information  that 

confirms  existing  belief  /  hypothesis 

•  Overgeneralization  -  generalizing  from  few  cases  or  from  non-representative  cases 

Implications  for  ABAIS  The  belief  assessment  component  of  ABAIS  corresponds  to 
the  situation  assessment  discussed  above,  in  that  the  currently  active  beliefs  and  knowledge 
schemas  influence  all  stages  of  the  situation  assessment  process.  The  assessment  of  a  user’s 
belief  state  thus  amounts  to  the  identification  of  his/her  set  of  knowledge  schemas  that  guide 
situation  assessment.  The  situation  assessment  literature  helps  identify  both  the  distinct  stages  of 
situation  assessment,  and  the  role  that  specific  knowledge  plays  in  this  process.  The  cognitive 
bias  literature  helps  identify  the  set  of  specific  performance  errors  that  can  result  from  cognitive 
biases.  Both  of  these  sources  provide  a  systematic  basis  for  identifying  possible  belief  states,  for 
analyzing  individual  history  information  and  applying  it  to  dynamic  belief  assessment,  and  for 
identifying  the  relationships  between  specific  affective  states  and  cognitive  biases. 

2.6  Enabling  Technologies 

This  section  summarizes  the  relevant  technologies,  including  wearable  computing, 
knowledge  elicitation  techniques,  rule-based  reasoning,  COTS  products  for  software 
development,  and  existing  systems  relevant  to  this  effort. 
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2.6.1  Wearable  Computers 

Wearable  computers  represent  one  of  the  latest  trends  in  portable  computing.  As  the  name 
implies,  ‘wearables’  are  distinguished  from  the  more  traditional  desktops,  laptops,  and  palmtops 
by  the  fact  that  they  can  actually  be  worn.  This  characteristic  implies  a  number  of  features 
regarding  weight  and  input/output  devices.  Wearable  computers  have  the  following 
characteristics  (Rhodes,  http://wearables.www.media.mit.edu/projects/wearables/'): 

•  Light-weight  and  unobtrusive  (possibly  embedded  in  clothing  or  even  jeweliy 

/http://www.rnedia.rnit.edu/affect/AC  research/projects/affective  iewelrv.html) 

•  Portable  while  operational,  allowing  computing  while  user  is  walking  or  going  about  other 

daily  activities 

•  Non-traditional  input-output  devices  (e.g.,  heads-up  displays,  chording  keyboard  input 

devices) 

•  Allowing  hands-free  use  by  facilitating  speech  input/output  and  heads-up  display 

•  Attention-capturing  mechanisms  allow  user  notification  when  events  of  interest  occur  (e.g.,  e- 

mail) 

Aside  from  their  size  and  portability,  the  key  distinguishing  features  of  wearables  are  the 
non-traditional  input-output  (I/O)  devices.  For  text  input,  the  Twiddler  chorded  keyboard  can  be 
used,  which  allows  rapid,  one-handed  text  input.  Additional  input  devices  include  video 
cameras,  microphones  for  speech,  and  a  variety  of  physiological  sensors  (see  also  discussion 
below).  For  output,  the  Private  Eye  LED  display  has  been  used  which  produces  a  monochrome 
720x780  image.  Wearable  processors  range  from  the  older  486  through  more  current  Pentium 
processors. 

Another  feature  of  wearables  is  the  presence  of  wireless  communication  devices,  for 
interconnecting  the  wearables  on  a  single  individual  (personal  area  network),  for  data  transfer 
between  individuals,  and  for  communicating  with  the  Web  and  mail  servers. 

The  applications  of  wearables  range  from  personal  assistants  which  facilitate  information 
management  and  Web  access,  through  job  aids  such  as  troubleshooting  and  maintenance 
(Columbia  University  KARMA  system),  to  individual  physiological  monitoring  systems  for 
military  applications  (BBN’s  Pathfinder). 

Because  of  their  unobtrusiveness  and  “perpetual  presence”,  wearable  computers  are  ideal  for 
the  type  of  physiological  signal  collection  necessary  for  reliable  affective  assessment.  The  ability 
to  collect  data  throughout  the  day  and  across  multiple  activities  facilitates  collection  of  the 
variety  of  baseline  physiological  measures  necessary  for  personalized  affective  assessment. 
Much  progress  has  been  made  in  the  wearable  technology  over  the  past  5  years,  both  in 
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commercial  settings,  and  in  university  laboratories.  Many  off-the-shelf  systems  are  available  for 
sensing  a  variety  of  physiological  signals.  For  example,  the  Polar  heart  rate  monitor  detects  heart 
rate  and  downloads  the  data  to  a  PC.  Its  cost  is  between  $150  and  $500.  Thought  Technologies 
ProComp  and  FlexComp  systems  provide  8  channels  A/D  converters  and  a  variety  of  sensors 
(e.g.,  BVP,  respiration,  heart  rate,  EKG,EEG,  EMG,  temperature)  which  can  be  interfaced  with  a 
PC.  The  cost  ranges  from  $1,500  to  $2,500  for  the  system  plus  about  $200  per  sensor.  Another 
off-the-shelf  system  is  Biopak  Systems’  MP100WSW  that  provides  an  interface  card  and  plug¬ 
ins  for  a  variety  of  amplifiers  and  sensors.  The  basic  system  is  $3,500,  and  sensors  are  between 
$100-200  each.  Sensors  include  the  standard  heart  rate,  EMG,  GSR,  skin  temperature,  blood 
pressure,  respiration,  EEG,  etc.  A  wireless  transmitter  is  also  available  for  up  to  12  feet. 

Laboratory  systems  include  MIT  Affective  Computing  Laboratory’s  galvanic  skin  response 
sensors  embedded  in  shoes,  rings,  and  bracelets.  GSR  provides  a  good  measure  of  arousal  and 
anxiety,  but  traditional  GSR  measures  are  often  impractical,  since  they  require  sensors  that  attach 
to  a  finger  and  thus  interfere  with  other  activities.  The  wearable  GSR  sensors  developed  at  MIT 
thus  promise  to  take  advantage  of  the  diagnosticity  of  this  signal,  while  eliminating  many  of  the 
practical  issues  that  prevented  its  application  with  more  traditional  sensors.  Other  systems 
developed  at  MIT  include  MicroOptical  eyeglasses  or  Private  Eye,  Hand-held  chording  keyboard 
(Twiddler),  and  ProComp  sensing  system  with  custom  sensors  (jewelry,  shoes,  PAN  (personal 
area  network). 

2.6.2  Knowledge  Elicitation  Techniques 

KE  techniques  have  been  developed  by  psychologists  and  artificial  intelligence  researchers  to 
access  human  knowledge  structures,  whether  in  the  context  of  memory  and  expertise  research 
(Olson  &  Biolsi,  1990),  or  in  the  more  applied  setting  of  knowledge-based  system  construction 
(Gaines  &  Boose,  1988)  and  mental  model  research  (Klein,  1989;  Rouse  &  Miller,  1986). 
Multiple  KE  methods  exist  and  a  number  of  variations  of  these  methods  have  been  described  in 
the  literature.  A  variety  of  KE  techniques  exist  and  have  been  extensively  described  in  the  KE 
literature  (Cooke,  1995).  In  spite  of  the  recent  proliferation  of  specific  KE  techniques,  the  basic 
methods  can  be  grouped  into  three  broad  categories:  direct,  indirect,  and  observational. 

Direct  techniques  are  based  on  the  assumption  that  the  subject  matter  experts  (SME’s) 
are  able  to  articulate  their  problem  solving  knowledge  in  response  to  direct  questions. 

•  Simple  interview ,  where  subjects  are  asked  a  series  of  questions  about  the  task.  Interviews  can 
vary  in  the  degree  of  structure  from  completely  open  ‘Tell  me  about  the  mission  planning 
process”  to  very  specific  “Why  did  you  decide  to  use  an  active  missile?” 
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•  Protocol  analysis  (Ericsson  &  Simon,  1984),  where  subjects  are  asked  to  think-aloud  while 

performing  a  task  and  these  data  are  later  analyzed  to  identify  knowledge  structure  and 
inferencing  processes. 

•  Interruption  analysis,  where  subjects  are  interrupted  during  a  critical  moment,  usually 

immediately  prior  to  or  after  an  important  decision  has  been  made,  and  is  asked  why  they 
made  a  particular  decision. 

•  Critical  decision  method  (Klein  et  al.,  1989),  a  form  of  retrospective  analysis  where  subjects 

are  presented  with  an  unusual  or  particularly  complex  incident  or  situation  and  asked  a 
series  of  questions  designed  to  elicit  factors  influencing  their  situation  assessment  and 
decision-making  processes. 

•  Inferential  flow  analysis,  where  subjects  are  asked  a  series  of  “why”  and  “how”  questions  to 

elicit  the  domain  causal  models  underlying  of  their  reasoning. 

While  the  direct  techniques  represent  methodologies  for  knowledge  elicitation  and 
knowledge  acquisition,  they  have  several  drawbacks  as  techniques  for  obtaining  the  expert's 
underlying  mental  models  and  details  of  the  reasoning  processes.  First,  only  data  accessible  to 
conscious  awareness  can  be  reported.  There  is  an  on-going  debate  as  to  whether  the  data  reported 
in  fact  represent  the  actual  underlying  thought  processes  or  whether  they  are  reconstructed  by  the 
expert  and  have  little  to  do  with  the  actual  mental  models  and  processes.1  Psychological 
literature  contains  many  experiments  reporting  exactly  such  reconstructions  (Nisbett  &  Wilson, 
1977)  .  There  is  evidence  that  truly  expert  knowledge  is  difficult  to  articulate  and  that  what  is 
being  reported  by  the  expert  is  intermediate  expertise  level  of  reasoning  (Schmidt,  Boshuizen  & 
Hobus,  1988;  Berry,  1987).  Second,  even  if  we  accept  introspection  as  a  reliable  means  of 
accessing  internal  processing,  direct  verbal  techniques  have  applicability  only  in  situations  where 
expert  problem-solving  is  verbally  mediated  or  at  least  when  it  can  be  expressed  in  terms  of 
language.  This  is  not  typically  the  case  for  tasks  which  rely  on  perceptual  and  motor  processing, 
which  are  often  performed  on  an  almost  reflexive  basis,  and  are  difficult,  if  not  impossible,  to 
articulate.  To  the  extent  that  such  processing  forms  the  basis  of  higher-level  reasoning,  such  as 
that  used  in  complex  tactical  pattern  recognition  in  battlefield  management,  a  different  set  of 
techniques  must  be  used  to  augment  the  purely  verbal  ones.  Third,  even  when  their  knowledge 
can  be  articulated  in  principle,  in  practice  experts  may  not  be  able  to  articulate  all  of  the 
reasoning  underlying  their  decision  making  when  asked  a  direct  question.  Finally,  experts  may 


1  Note  that  when  we  question  whether  data  are  accessible  via  introspection  we  are  speaking  here  about  the 
detailed  mental  models  and  reasoning,  not  about  the  basic,  general  knowledge  of  the  task  and  the  domain,  which 
can  clearly  be  articulated  and  obtained  via  interviews. 
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not  be  willing  to  express  their  knowledge,  due  to  fear  of  “giving  away”  private  knowledge  or, 
sometimes,  embarrassment  because  of  the  idiosyncratic  nature  of  this  knowledge. 

To  address  these  problems,  indirect  techniques  have  been  developed.  These  techniques 
do  not  rely  on  the  assumption  that  all  expert  knowledge  can  be  readily  articulated  in  response  to 
direct  questions  and  specific  probes.  The  indirect  methods  attempt  to  by-pass  this  limitation  by 
accessing  elements  of  the  internal  structures  through  a  series  of  simpler  questions,  for  example, 
obtaining  lists  of  similarities  and  differences  among  salient  domain  entities.  The  more  complex 
domain  structures  and  basis  for  inferencing  are  then  reconstructed  from  these  knowledge 
elements.  These  techniques  thus  do  not  rely  on  simple  introspection,  nor  are  they  limited  to  data 
that  are  easily  verbalized.  This  latter  attribute  is  particularly  important  in  domains  where 
visualization-based  problem  solving  is  critical,  such  as  battlefield  situation  assessment. 

Indirect  methods  include  a  variety  of  proximity  scaling  techniques ,  which  first  elicit 
similarity  ratings  among  domain  entities  (e.g.,  “On  a  scale  of  1  to  10,  how  similar  are  COA  1  and 
COA  2?”)  and  from  these  data  construct  a  specific  representational  structure  using  some  type  of 
multivariate  statistical  analysis  method.  The  structures  include  graphs  (Pathfinder  algorithm 
(Cooke  et  al.,  1987)),  hierarchies  (hierarchical  clustering  (Johnson,  1967)),  or  a  plots  in  a 
continuous  multi-dimensional  space  (multi-dimensionsal  scaling  (Shiffman  et  al.,  1981)).  The 
nodes  in  these  representations  represent  the  domain  entities  and  the  connecting  links  or  spatial 
distribution  represents  some  relation  among  these  entities.  The  structures  can  then  be  analyzed 
further  to  identify  desired  domain  characteristics.  While  powerful  psychological  research 
instruments,  proximity  scaling  methods  are  not  always  practical  for  KE,  due  to  the  time,  effort, 
and  complex  software  required. 

A  powerful  alternative  technique  is  repertory  grid  analysis  (RGA),  a  method  adapted  from  a 
psychological  theory  of  personality  and  cognition  (Kelly,  1955)  and  widely  used  in  clinical 
settings.  RGA  is  based  on  eliciting  similarities  and  differences  among  salient  domain  entities 
(e.g.,  “List  all  similarities  and  differences  between  COA  1  and  COA  2.”).  RGA  consists  of  three 
steps.  First,  domain  entities  are  identified  (these  form  the  column  labels  of  the  repertory  grid 
matrix).  Second,  the  entities  are  compared  two  or  three  at  a  time  to  obtain  similarities  and 
differences  (these  form  the  row  labels  of  the  repertory  grid  matrix).  Third,  each  entity  is  rated 
along  each  attribute  on  some  scale  (e.g.,  “On  a  scale  of  1  to  10,  how  ‘risky’  is  COA  1?”).  The 
filled-in  grid  can  then  be  analyzed  via  a  number  of  methods,  including  all  of  the  methods  used 
for  proximity  scaling  techniques. 
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2.6.3  Rule-Based  Reasoning 

Rule-based  reasoning  (RBR)  is  an  established  method  of  knowledge-based  inferencing.  An 
RBR  system  consists  of  a  knowledge-base,  represented  in  terms  of  if-then  production  rules;  data 
or  facts  describing  the  current  situation,  represented  in  terms  of  variables  matched  against  the 
rule  antecedents;  and  an  inference  engine,  which  performs  the  fact  /  rule-base  matching,  selects 
the  highest  matching  rules,  and  instantiates  these  rules  and  applies  their  consequents.  Rule- 
based  systems  can  vary  along  several  dimensions  including  the  type  of  inferencing  (forward 
chaining  vs.  backward  chaining),  the  type  of  matching  strategies  (fuzzy  vs.  exact  matches),  and 
the  type  of  control  implemented  through  the  rule-selection  process  (breadth,  depth,  or  heuristic). 

2.6.4  COTS  Products 

The  ABAIS  workstation  demonstration  prototype  was  developed  using  COTS  packages  as 
much  as  possible  to  minimize  the  implementation  effort.  A  brief  description  of  the  tools  used  for 
the  Phase  I  effort,  as  well  as  relevant  tools  for  Phase  II,  are  included  below. 

JAVA  JAVA  was  selected  as  the  core  development  language.  JAVA  is  a  general  purpose, 
object-oriented,  high-level  programming  language,  similar  to  C++,  but  having  several 
advantages  over  C++.  First,  JAVA  uses  a  syntax  simpler  than  C++,  while  still  maintaining  the 
pure  object-oriented  paradigm.  JAVA  provides  a  large  number  of  GUI  primitives  and  there  is 
large  number  of  JAVA-based  GUI  builder  tools.  In  addition,  the  increasing  size  of  the  JAVA 
user  group  provides  a  good  source  of  freeware  GUI  elements.  JAVA  supports  good  cross¬ 
platform  compatibility.  JAVA  also  supports  links  to  VRML  for  future  embedding  within  virtual 
reality  environments. 

Borland  Jbuilder  2.0  Borland's  JBuilder  2.0  was  selected  as  the  development 
environment  to  support  rapid  design  and  development  of  GUI's.  JBuilder  was  selected  because  it 
provides  one  of  the  best  available  GUI  design  environments  and  offers  a  full  suite  of  JAVA 
objects. 

JESS  JESS  is  a  rule-based  system  shell  that  implements  the  NASA-developed  CLIPS 
public  domain  rule-base  system  within  JAVA.  JESS  will  be  used  to  develop  the  rule-based 
reasoning  components  of  the  individual  ABAIS  modules. 

Hngin  is  a  COTS  product  supporting  the  development  of  Bayesian  belief  nets.  Hugin  has  a 
full  GUI,  supporting  the  graphical  construction  and  execution  monitoring  of  belief  nets. 
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2.6.5  Existing  Relevant  Systems 

VIEW  is  a  knowledge  elicitation  workstation  developed  at  Charles  River  Analytics 
(Hudlicka  et  al.,  1999)  which  provides  an  integrated  toolkit  for  interactive,  visualization-based 
knowledge  elicitation.  VIEW  supports  both  direct  KE  methods  (e.g.,  protocol  analysis,  critical 
decision  method,  inferential  analysis,  etc.)  and  indirect  methods  (e.g.,  repertory  grid  analysis). 
VIEW  facilitates  the  construction  of  KE  scripts  designed  to  administer  any  combination  of 
specific  KE  techniques.  VIEW  is  well-suited  for  supporting  off-line  knowledge  elicitation  effort 
required  to  identify  the  factors  necessary  for  both  affect  and  belief  assessment. 
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3.  System  Description 

This  chapter  provides  a  description  of  the  ABAIS  prototype.  Section  3.1  describes  the  system 
architecture  and  provides  a  general  overview  of  system  functionality.  Sections  3.2  through  3.5 
then  describe  the  four  key  modules  of  the  ABAIS  prototype:  user  state  assessment  (section  3.2), 
impact  prediction  (section  3.3),  strategy  selection  (section  3.4),  and  graphical  user  interface  / 
decision  support  system  (GUI/DSS)  adaptation  (section  3.5).  Section  3.6  describes  the  ABAIS 
GUI  and  section  3.7  describes  the  simulation  component. 

3.1  System  Architecture  and  Functional  Overview 

The  ABAIS  architecture  implements  the  adaptive  methodology  summarized  in  table  3.1-1 
and  consists  of  four  modules:  1)  user  state  assessment,  which  identifies  the  user's  affective  state 
and  task-relevant  beliefs;  2)  impact  prediction,  which  identifies  the  effect  user  state  on 
performance;  3)  strategy  selection,  which  selects  a  compensatory  strategy;  and  4)  GUI  /DSS 
adaptation,  which  modifies  the  user  interface  content  and  format  to  enhance  detection, 
recognition,  and  assimilation  of  incoming  data;  that  is,  to  enhance  situation  awareness.  The 
ABAIS  framework  architecture  implementing  this  methodology  is  shown  in  figure  3.1-1.  Since 
no  single  reliable  method  currently  exists  for  affect  assessment,  the  ABAIS  workstation  design 
provides  facilities  for  the  flexible  combination  of  multiple  methods  of  user  state  assessment  (e.g., 
diagnostic  tasks,  self  reports,  multiple  physiological  measures,  individual  training  and  history, 
etc.). 

The  original  plan  called  for  integration  of  ABAIS  with  the  Charles  River  Analytics  SD_PVI 
system  (Mulgund  et  al.,  1997).  However,  it  was  determined  that  due  to  platform  and  software 
incompatibilities,  as  well  as  the  focus  of  the  current  effort,  the  ABAIS  demonstration  was  better 
served  through  a  custom-coded  GUI  and  simulation  environment.  The  SD_PVI  system  from 
Charles  River  Analytics  thus  served  primarily  as  a  model  for  the  flight  dynamics  required  for  the 
flight  simulation. 

ABAIS  Run-Time  Execution  Modes  The  core  ABAIS  framework  is  integrated 
within  a  dynamic  flight  simulation  environment  and  supports  two  modes  of  system  operation:  1) 
pilot-as-user  mode,  where  the  user  actually  flies  the  aircraft  and  interacts  with  a  simulated 
environment  consisting  of  other  friendly  aircraft,  enemy  aircraft,  radars,  and  weapons;  and  2) 
analyst-as-user,  where  the  analyst  watches  a  simulation  of  a  scripted  task  and  monitors  the 
(scripted)  pilot's  performance,  and  the  system  run-time  performance  (i.e.,  results  of  the  rule- 
based  inferencing). 
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Table  3.1-1:  Methodology  for  Proposed  Adaptive  Framework 


METHODOLOGY  STAGE 

EXAMPLE 

1.  ASSESS  USER  STATE 

•  Identify  affective  state 

•  Identify  individual  beliefs 

•  Affective  state:  Anxious 

•  Belief:  Belief  that  approachinq  aircraft  are  hostile 

2.  PREDICT  IMPACT 

(generic  &  task  specific) 

•  Predict  impact  of  affective  states 

•  Predict  impact  of  individual  beliefs 

Generic 

•  Anxiety  Impacts  Attention  &  Perception 

-  Increased  focus  on  threatening  stimuli 

-  Perception  of  ambiguous  signals  as  threats 

•  Indiv.  history  creates  biases  in  expectations  of  outcome 
Specific 

•  Focus  on  threat  detection  &  weapons  deployment, 
neglecting  communication  info  from  wingman 

•  Bias  towards  interpreting  unknown  returns  as  threats 

3.  SELECT  STRATEGY 

•  Select  compensatory  strategies  to 
prevent  biased  performance 

•  Present  reminders  to  prevent  neglect  of  tasks 

•  Present  broader  evidence  to  counteract  threat-estimation 
bias 

•  Present  contrary  evidence  to  counteract  failure-driven 
confirmation  bias 

4.  ADAPT  AIDING  INFO  TYPE  & 
PRESENTATION 

•  Provide  additional  data  about  ambiguous 
signals 

•  Select  info  format  enhancing  detection 
and  assimilation 

•  Direct  an  associated  DSS  to  present  additional  data  about 
ambiguous  signals 

•  Present  explicit  estimates  of  times  required  for 
task  completion  to  prevent  premature  engagement 

•  Use  multi-modal,  customized  attention-capturing 
presentation  to  assure  detection 

ABAIS  ADAPTIVE  FRAMEWORK 


USER  STATE  ASSESSMENT 


Affect  Assessment 

-  Physiological 

-  Diagnostic  tasks 

-  Self  report 

-  KB  methods 


Belief  Assessment 

-  KE  techniques 

-  KB  methods 

-  Diagnostic  tasks 


IMPACT  PREDICTION 

GENERIC  TASK 
SPECIFIC 


specific  affect/ 
belief  induced 
impact  on  task 
performance 


STRATEGY  SELECTION 


task-specific 

compensatory' 


Task  Simulation 


DECISION  SUPPORT 
SYSTEM 


Figure  3.1-1:  ABAIS  Architecture  Implementing  the  Adaptive  Methodology 
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Both  modes  include  a  pilot  GUI,  which  simulates  key  cockpit  displays  relevant  to  the  task: 
the  HUD,  a  radar  display  showing  all  sensor  data,  and  two  windows  showing  incoming 
communications  and  alarms,  respectively.  To  support  these  two  modes  of  operation,  the  ABAIS 
simulation  environment  supports  both  human-in-the-loop  flight  simulation,  and  script-based 
agent  simulation.  The  design  includes  a  script  definition  functionality  which  allows  the 
interactive  graphical  specification  of  customized  scripts  via  a  variety  of  pop-up  menus  associated 
with  the  individual  cells.  Figure  3.1-2  shows  a  segment  of  the  task  script  matrix  for  the 
demonstration  task. 

User  Profiles  During  a  given  run,  ABAIS  uses  pilot  profiles  defining  critical  features  and 
information  required  for  the  affect  and  belief  assessment,  and  for  adaptation  strategy  selection 
and  implementation.  User  profiles  include  information  about  the  pilot’s  personality  (e.g., 
anxiety-tolerant  vs.  anxiety-prone),  training  and  proficiency  (e.g.,  experienced  vs.  novice), 
individual  history  (e.g.,  recent  failures  or  successes),  etc.  This  information  forms  the  basis  of 
the  dynamic  affect  and  belief  assessment  during  a  scenario  run.  User  profiles  also  include 
information  about  the  individual  pilots’  preferences  for  information  presentation,  which  allows 
customized  adaptations  based  on  the  pilot’s  specific  information  processing  preferences  (e.g., 
visual  vs.  auditory  modality  for  alarms,  specific  colors  and  attention  capture  graphics,  etc.).  The 
highly  individualized  profiles  represent  a  critical  component  of  the  adaptive  methodology,  and 
provide  the  necessary  background  information  that  allows  the  assessment  of  the  pilot’s  current 
affective  state  and  likely  belief,  during  the  course  of  the  task. 

ABAIS  Demonstration  Task  ABAIS  functionality  was  demonstrated  in  the  context  of 
a  sweep  task,  focusing  on  adaptation  with  respect  to  a  single  user  (i.e.,  a  single  pilot).  The 
primary  ABAIS  demonstration  used  the  analyst-as-user  mode,  due  to  the  precise  coordination 
required  to  "fly"  the  sweep  task  demonstration  script  and  simulate  the  desired  sequence  of  events 
to  demonstrate  real-time  adaptation.  In  this  mode,  the  aircraft  is  controlled  by  a  script  that 
simulates  the  pilot  behavior  (e.g.,  “centering  the  dot”  on  a  HUD)  and  the  dynamic  task 
environment  (e.g.,  changes  in  radar  contact  status  due  to  incoming  IFF  data).  The  pilot-as-user 
(human-in-the-loop)  mode  was  demonstrated  separately,  during  an  evaluation  of  physiological 
user  state  assessment.  This  evaluation  involved  the  design  and  development  of  a  wearable  heart 
rate  sensor,  whose  output  is  linked  directly  into  the  ABAIS  GUI  /  DSS  Adaptation  module  to 
provide  GUI  modifications  in  response  to  specific  changes  in  the  user’s  heart  rate.  The 
successful  implementation  of  this  sensor  demonstrated  the  feasibility  of  using  real-time 
physiological  data  to  drive  the  adaptation  logic. 

The  ABAIS  Testbed  Environment  A  key  objective  of  the  overall  effort  is  to  develop 
a  flexible  testbed  for  the  exploration  of  multiple  affective  assessment  methods  and  adaptation 
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strategies.  To  support  this  objective,  we  developed  an  analyst  GUI  enabling  the  flexible 
specification  of  background  information  about  the  pilot,  selection  of  run  time  parameters 
controlling  the  simulation,  and  system  performance  monitoring.  The  analyst  GUI  displays 
include  a  variety  of  windows  containing  dialog  boxes  for  defining  the  background  information 
necessary  for  the  knowledge-based  user  assessment  (e.g.,  pilot  preference  profile,  pilot 
individual  history);  dialog  boxes  for  specifying  the  system  run  time  parameters  (e.g., 


Figure  3.1-2:  Segment  of  a  Script  Matrix  Defining  the  Demonstration  Task 
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what  windows  should  be  visible  for  monitoring  system  performance);  and  windows  allowing 
monitoring  of  system  performance  (e.g.,  rule  base  and  rule  instantiation,  physiological  data, 
inferred  pilot  affective  state). 

The  remainder  of  this  section  describes  in  detail  the  individual  modules  of  the  core  ABAIS 
architecture  (sections  3.2  through  3.5),  the  graphical  user  interface  (section  3.6),  and  simulation 
(section  3.7). 

3.2  User  State  Assessment  Module 

The  overall  functionality  of  this  module  is  summarized  in  figure  3.2-1.  The  User  State 
Assessment  Module  receives  a  variety  of  data  about  the  user  and  the  task  context  and  from  these 
data  identifies  the  user's  predominant  affective  state  and  individual  beliefs  and  preferences,  as 
they  relate  to,  and  potentially  influence,  task  performance.  The  assessment  methods  are 
discussed  in  detail  below. 


Figure  3.2-1:  Functionality  of  the  User  State  Assessment  Module 


3.2.1  Affective  Assessment 

Since  no  single  reliable  method  currently  exists  for  affective  assessment,  the  User 
Assessment  module  provides  facilities  for  the  flexible  combination  of  multiple  methods.  A 
number  of  such  complementary  methods  exist.  These  include: 

•  Sensing  of  physiological  signals  (e.g.,  galvanic  skin  response  (GSR),  heart  rate,  heart  rate 

variability,  blood  volume  pressure,  muscle  tension,  etc.); 

•  Recognizing  the  affective  state  from  facial  expression  or  pupil  size  changes; 

•  Use  of  diagnostic  tasks  (e.g.,  test  tasks  such  as  the  detection  rate  and  interpretation  accuracy 

of  a  particular  sequence  of  stimuli); 

•  Self-reports  (i.e.,  subjective  answers  to  specific  questions  about  the  user's  state,  such  as  those 

used  in  workload  estimation);  and 

•  Use  of  AI  knowledge-based  methods  (e.g.,  rule-based  reasoning  (RBR),  CBR  (case-based 

reasoning),  belief  nets,  etc.))  to  derive  likely  affective  state  based  on  factors  from  current 
task  context,  personality  traits,  and  individual  history. 

Each  of  these  methods  has  its  associated  advantages  and  disadvantages,  and  none  alone  is 
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currently  sufficient  for  reliable  affective  state  identification.  The  key  to  successful  affect 
assessment  is  therefore  the  coordinated  use  of  multiple  methods.  The  overall  ABAIS  framework 
thus  calls  for  the  coordinated  use  of  all  of  these  methods,  to  improve  reliability  of  the  affective 
state  assessment. 

For  the  Phase  I  effort,  we  focused  on  a  knowledge-based  approach,  which  combines  data 
from  all  categories  of  user  information  and  allows  the  demonstration  of  the  feasibility  of  the 
overall  adaptive  framework.  Under  a  Phase  II  effort,  a  broader  scope  of  methods  would  be 
implemented,  including  limited  physiological  sensing,  self-reports,  and  diagnostic  tasks.  Under 
the  Phase  I  effort,  the  primary  focus  of  demonstration  was  anxiety  assessment  and  adaptation, 
with  a  secondary  focus  on  the  assessment  of,  and  adaptation  to,  a  heightened  state  of 
obsessiveness,  resulting  from  a  combination  of  increased  anxiety  and  a  specific  personality  trait 
(conscientiousness  from  the  “Big  5”  factor  group).  Under  a  Phase  II,  a  broader  range  of 
affective  states  and  personality  traits  would  be  assessed,  although  the  primary  focus  would 
continue  to  be  on  anxiety.  Anxiety  was  selected  both  because  it  is  the  most  prevalent  affect 
during  crisis  situations,  and  because  its  influence  on  cognition  has  been  extensively  researched, 
and  empirical  data  exist  to  support  specific  impact  prediction  and  adaptation  strategies. 

A  variety  of  factors  contribute  to  an  individual’s  affective  state  and  the  expression  of 
particular  personality  traits  (see  figure  3.2. 1-1).  Examples  of  such  specific  factors,  in  the  context 
of  the  Phase  I  demonstration  task,  are  shown  in  table  3.2. 1-1.  The  user  assessment  module  uses 
knowledge  from  each  of  these  categories  of  factors  to  derive  the  user's  affective  state  or  relevant 
personality  trait,  using  a  rule-based  reasoning  approach  (see  figure  3.2. 1-2).  The  assessment 
process  implements  a  fuzzy  rule-based  approach  consisting  of  four  stages.  First,  a  user  profile  is 
specified  in  terms  of  static  and  dynamic  data,  representing  task-relevant  factors  about  the  user 
(see  table  3.2.1-2  and  figure  3.2.1-3,  which  shows  the  ABAIS  dialog  boxes  for  user  profile 
specification).  Second,  the  data  in  this  profile  are  matched  against  the  rules  in  the  user 
assessment  rule-base,  following  the  standard  rule-based  reasoning  algorithms  and  using  fixed 
weights  for  conflict  resolution  (see  figure  3.2. 1-4  for  examples  of  rules).  Third,  each  relevant 
factor,  represented  by  an  instantiated  rule,  contributes  a  numerical  weight  component  to  the 
overall  score  of  the  corresponding  affect.  (This  is  the  anxiety  weight  factor,  or  AWF,  shown  in 
the  rules.)  Individual  factors  or  categories  of  factors  may  be  weighted  differently,  to  reflect  their 
differential  influence  on  the  overall  affective  state  (e.g.,  static  task  factors  will  typically  have  a 
lower  weight  than  dynamic  factors  and  real-time  physiological  signals).  Finally,  after  all  relevant 
rules  are  instantiated,  the  overall  anxiety  level  is  computed  and  the  resulting  value  is  mapped 
onto  a  three-valued  qualitative  variable  indicating  a  low,  medium,  or  high  anxiety  level. 
Examples  of  specific  factors  in  each  of  the  relevant  categories  are  discussed  below. 
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task  context 

(e.g.,  type,  complexity, 
time  of  day,  length  of  task)^S. 

external  events  \ 

(e.g.,  equipment  failures, 
enemy  configuration, 
team  effects) 

personality 

(e.g.,  anxiety  proneness,  _ _ 

negative  emotionality, 
extraversion  /  introversion, 
physiological  reactivity) 

individual  history 

(e.g.,  past  failures  /  successes, 
current  affective  state,  reactions  to 
specific  events  /  people)  > 

training  ^ 

(e.g.,  competencies  in  specific  / 
procedures  /  contexts)  / 

physiological  data 

(e.g., heart  rate,  variability,  etc.) 


INFERRED  CURRENT 
AFFECTIVE  STATE 


weights  can  be 
associated  w/  different 
sources  of  data 


Figure  3. 2. 1-1:  Sources  of  Information  for  Deriving  Pilot's  Affective  State 

Table  3. 2. 1-1:  Examples  of  Task  Specific  Factors  for  Contributing  to  the 

Pilot's  Affective  State 


External  Event  Factors 

Equipment  failures 

=  {engine,  nozzle  indicator,  IFF,  RWR,  NCTR,  EID} 

Target  distance 

==  {not  detected  1  BVR  1  WVR} 

Target  distance  change 

==  {closing  1  opening} 

Firing  status 

==  {cleared  to  fire  1  not  cleared  to  fire} 

NCTR  output 

=  {none  1  unknown  1  friendly  1  hostile} 

IFF  output 

==  {friendly  1  no  response} 

RWR  output 

—{air-to-air  unknown  1  air-to-air  hostile  1  SAM } 

Number  of  hostile  a/c 

=={2  141  6  l>6} 

Number  of  unknown  a/c 

==  {2  1  4  1  6  1  >  6} 

Active  radar  output 

=  {unknown  l  hostile  I  friendly} 

Individual  History  Factors 

Successful  situations  in  past 

:==  {Planning  1  Detection  1  Commitment  1  Sorting  1  Engagement  1  Egress  1  Evasion  1  Hit 

I  enemy  radar  lock  -  evasion  I  enemy  radar  lock  -  hit  I  unknown  radar  lock  - 
evasion  1  unknown  radar  lock  -  hit} 

Failure  situations  in  past  :==  {Planning  I  Detection  I  Commitment  I  Sorting  I  Engagement  I  Egress  I  Evasion  I 

Fratricide  I  enemy  radar  lock  -  evasion  I  enemy  radar  lock  -  hit  I  unknown  radar  lock 
-  evasion  I  unknown  radar  lock  -  fratricide  repeated  unsuccessful  IFF  interrogations 
followed  by  fratricide} 

Affective  reactions  to  specific  events 

:==  (unknown  NCTR  type  =  (anxiety  weight  factor=8 
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Affective 

State 

(low,  med,  hi) 


Figure  3.2. 1-2:  Rule-Based  Approach  to  Affect  Assessment 


Table  3.2. 1-2:  User  Profile  Example: 
Anxiety  Prone  (High-Anxious)  Pilot  Profile 


Personality  factors 

Emotional  stability  =  low 

Training  /  Proficiency  factors 

air  combat  hours  =  low 

air  combat  exercises  hours  =  medium 

training  hours  =  low 

recency  of  air  combat  =  low 

recency  of  air  combat  exercises  =  low 

recency  of  training  =  medium 

Specific  events  in  individual  history  contributing  to  anxiety 

Cleared  to  fire 
Unknown  type  from  NCTR 
Unsuccessful  IFF  interrogation 
Unknown  RWR  radar  lock 
SAM  lock 
"Centering  the  dot" 


50 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Figure  3.2. 1-3:  Dialog  Boxes  for  the  Interactive  Specification 
of  User  Profile  Information 
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^Affect  Assessment  Rule  Base 


[F  (unknown  type  from  NCTR) 

THEN  (anxiety  weightfactor=  8) 

IF  (cleared  to  fire) 

THEN  (anxiety  weight factor=  8) 

IF  (coordination  w ■/  wingman  =  poor  |  neutral  |  good) 

THEN  (AWF  =  5 1  0  | -5) 

IF  (#  hostile  targets  =  0  |  2  1 4  |  >  4  |  unknown) 

THEN  (anxiety  weightfactor=  0 1 3 1 6 1 9 1 8) 

IF  (#  of  contacts  increases) 

THEN  (anxiety  weight factor=  6) 

IF  (hostile  target  distance  =  not  detected  |  BVR  |  WVR) 

THEN  (anxiety  weight  factors  0  |  5 1 7) 

IF  (distance  change  of  hostile  targets  =  closing  |  opening) 
THEN  (anxiety  weight  factors  5 1  0) 

IF  (IFF  outputs  friendly  |  no  response} 

THEN  (anxiety  weight  factors  -1 0  1 1 0) 

IF  (RWR  outputs  air-to-air  unknown  |  air-to-air  hostile  |  SAM) 

TUICM  fanviofc/  for.+ni'  —  7  I  A  H  I 


WS 


Figure  3.2. 1-4:  Examples  Rules  Using  External  Events  Factors  to 

Derive  User  Affective  State 

The  factors  contributing  to  the  pilot’s  affective  state  fall  into  two  broad  categories:  static  and 
dynamic.  The  static  factors  represent  influences  that  remain  constant  throughout  the  task,  that  is, 
the  background  level  of  task  difficulty,  pilot's  training  and  proficiency  level,  pilot's  personality 
characteristics,  and  pilot's  individual  history.  The  values  of  these  factors  are  specified  prior  to  a 
particular  simulation  and  the  resulting  affect  weight  factors  is  calculated  once,  before  the 
simulation  begins.  In  contrast  to  this,  the  values  the  dynamic  factors  represent  the  changing 
external  environment  (e.g.,  incoming  data  from  sensors  such  as  radar  contacts)  as  well  as  the 
changing  state  of  the  pilot  (e.g.,  changes  in  particular  physiological  signals  such  as  heart  rate). 
These  values  are  provided  to  the  affective  assessment  rules  throughout  course  of  the  simulation, 
allowing  dynamic  computation  of  the  affect  weight  factor. 

For  the  Phase  I  effort  only  first-order  effects  were  represented;  in  other  words,  contributions 
of  individual  factors  to  an  overall  affective  state  (e.g.,  anxiety)  score  were  calculated 
independently. 

Specific  factors,  their  corresponding  values,  and  example  rules  using  these  factors  to  derive 
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an  affective  state  are  discussed  below.  It  is  important  to  keep  in  mind  that  the  factors,  their 
values,  and  the  corresponding  rules  are  specified  in  the  context  of  a  specific  task  and  can  be 
changed  depending  on  the  particulars  of  the  tasks  and  the  individual.  In  fact,  a  key  feature  of  the 
ABAIS  system  design  is  precisely  this  ability  to  modify  these  factors,  their  values,  and  their 
weights,  to  allow  individualized  adjustment  of  system  performance.  Such  individualized  system 
calibration  is  key  to  successful  adaptation,  given  the  large  space  of  possible  influences  of 
affective  and  belief  states  on  performance. 

Static  Factors  Task  context  factors  define  the  overall  difficulty  of  the  task.  These  are  in 
effect  the  static,  a  priori  factors  that  contribute  to  the  overall  level  of  task  difficulty  and  thus 
exert  a  potential  indirect  effect  on  the  overall  affective  state  (e.g.,  anxiety).  The  specific  values 
are  obtained  from  experts  via  direct  knowledge  elicitation  methods  and  can  be  further 
customized  to  reflect  the  particular  user's  experience  (i.e.,  for  a  particular  pilot  the  subjective 
complexity  for  specific  task  types  may  vary,  etc.).  Examples  of  these  factors,  and  their  possible 
values  in  the  context  of  the  Phase  I  demonstration  task,  are  summarized  in  table  3.2. 1-3. 
Examples  of  rules  using  these  factors  to  derive  an  affective  state  are  shown  in  table  3.2.1-4. 

Table  3.2. 1-3:  Summary  of  Task  Context  Factors  Used  During  User  State 

Assessment 


Type_l 

=  {Offensive,  Defensive) 

Type_2 

=  {air-to-air,  air-to-ground) 

Type_3 

=  {intercept,  CAP,  sweep) 

Phase 

=  {Planning,  Detection,  Commitment,  Sorting,  Engagement,  Egress) 

Complexity 

:=  {1-10} 

Weather 

:=  {clear,  cloudy,  fog,  precipitation) 

Light  conditions 

:=  {light,  dusk,  dark) 

Table  3.2. 1-4:  Examples  Rules  Using  Task  Context  Factors  to 
Derive  User  Affective  State 


IF  (phase  =  Planning  I  Detection  I  Commitment  I  Sorting  I  Engagement  I  Egress) 

THEN  (anxiety  weight  factor  =1  1 3 1 5  I  7 1 10 1 3) 

IF  (task  type_2  =  offensive  I  defensive)  THEN  (anxiety  weight  factor  =  5110) 

IF  (task  complexity  =  low  I  med  I  high)  THEN  (anxiety  weight  factor  =  1/5/10) 

IF  (weather  =  clear  I  cloudy  I  fog  I  precipitation  THEN  (anxiety  weight  factor  =  0121313) 

IF  (light  conditions  =  light  I  dusk  I  dark) _ THEN  (anxiety  weight  factor  =  01312) _ 

Training  and  proficiency  factors  represent  the  individual's  training  and  educational 
background,  as  well  as  the  current  proficiency  level.  Training  refers  to  the  total  number  of  flight 
hours,  combat  experience  and  combat  exercises,  as  well  the  specific  type  of  training  (e.g.,  type  of 
aircraft  or  simulator,  specific  types  of  missions,  etc.).  Skill  level  or  proficiency  depend  on  recent 
experience  or  "currency",  which  is  a  function  of  type  and  frequency  of  recent  training  (e.g.,  in 
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simulator  vs.  during  actual  exercises  or  combat  missions)  and  actual  tasks.  All  of  these  factors 
influence  the  overall  skill  level  and  specific  competencies  and  shortcomings  in  performance. 
Skill  level  and  proficiency  are  particularly  critical  in  assessing  the  level  of  automaticity,  which 
determines  the  degree  to  which  task  performance  may  degrade  under  conditions  of  heightened 
anxiety.  The  selected  training  and  proficiency  factors,  and  their  possible  values  in  the  context  of 
the  Phase  I  demonstration  task,  are  summarized  in  table  3.2. 1-5.  Examples  of  rules  using  these 
factors  to  derive  an  affective  state  are  shown  in  table  3.2. 1-6. 


Table  3.2.1-5:  Summary  of  Training  and  Proficiency  Factors  for  User  State 

Assessment 


Flight  hours  in  current  aircraft: 

Air  combat  :=  {low  1  medium  1  high} 

Air  combat  exercises  :=  {low  1  medium  1  high} 

T  raining  :=  {low  1  medium  1  high} 

Recency  of 

Air  combat  :=  {low  1  medium  1  high} 

Air  combat  exercises  :=  {low  1  medium  1  high} 

T  raining  :=  {low  1  medium  1  high} 

Specific  tasks  and  skill  level  :==  {radar  1  instrument  flyina 

dogfightino) 

Table  3.2. 1-6:  Examples  Rules  Using  Training  and  Proficiency  Factors 

to  Derive  User  Affective  State 

IF  (air  combat  hours  =  low  1  med  1  high) 

THEN  (AWF  =  61310) 

IF  (air  combat  exercises  hours  =  low  1  med  1  high) 

THEN  (AWF  =71410) 

IF  (training  hours  =  low  1  med  I  high) 

THEN  (AWF  =  41210) 

IF  (recency  of  air  combat  hours  =  low  1  med  1  high) 

THEN  (AWF  =  91510) 

IF  (recency  of  air  combat  exercises  hours  =  low  1  med  1  high) 

THEN  (AWF  =71410) 

IF  (recency  of  training  hours  =  low  1  med  1  high) 

THEN  (AWF  =61310) 

Personality  factors  represent  a  relatively  stable  set  of  characteristics  of  the  user's  personality 
that  may  contribute  to  his/her  affective  state  (e.g.,  emotional  stability  correlates  with  anxiety 
tolerance)  and  may  influence  behavior  in  general  (e.g.,  obsessiveness,  aggressiveness).  The 
selection  of  the  specific  personality  factors  below  was  guided  by  the  following  criteria: 

•  Empirical  evidence  for  existence  of  factor  as  a  distinct  personality  characteristic 

•  Empirical  evidence  or  knowledge  elicitation  data  indicating  specific  effects  of  the  personality 

factor  on  performance,  particularly  in  the  context  of  aviation  and  air  combat  tasks 

•  Likelihood  of  the  personality  factor  playing  a  role  in  the  selected  demonstration  task 

A  number  of  studies  exist  assessing  the  pilot  population  in  terms  of  a  variety  of  standard 
psychological  assessment  instruments  (e.g.,  NEO-PI-R,  MMPI,  etc.)  (Callister  et  al.,  1997),  as 
well  as  instruments  specialized  for  the  fighter  pilot  population  (e.g.,  ALAPS  (Retzlaff  et  al.. 
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1997)).  However,  little  systematic  empirical  work  has  been  done  in  the  general  area  of  linking 
personality  factors  to  specific  performance  influences  and  biases,  at  a  level  of  analysis  that 
would  provide  the  type  of  detail  necessary  for  real-time  adaptation.  This  type  of  research,  while 
critical  for  improved  understanding  of  pilot  performance,  is  both  methodologically  difficult,  and 
somewhat  controversial  in  nature. 

The  selection  of  the  ABAIS  personality  factors  was  therefore  based  on  general  personality 
research  results  (e.g.,  “Big  5’,  “Giant  3”  -  see  section  2.4  above),  and  on  knowledge  elicited 
from  domain  experts,  either  pilots  or  USAF  psychologists  and  scientists.  The  objective  was  to 
capture  personality  traits  that:  1)  are  likely  to  exist  in  the  pilot  population,  and  2)  exert  a 
pronounced  influence  on  behavior  during  the  performance  of  the  demonstration  task.  The 
selected  personality  factors,  and  their  possible  values  in  the  context  of  the  Phase  I  demonstration 
task,  are  summarized  in  table  3.2.1-7.  Emotional  stability  was  the  primary  factor  of  interest 
under  the  Phase  I  effort,  since  it  is  this  factor  that  correlates  with  anxiety  tolerance.  The  rule 
capturing  this  influence  in  shown  in  table  3.2. 1-8. 

Table  3.2. 1-7:  Summary  of  Personality  Factors  for  User  State  Assessment 


Emotional  stability 

:==  {1-10} 

Impulsiveness 

:=  <1-10} 

Risk  tolerance 

:=  {1-10} 

Aggressiveness 

:=  {1-10} 

Conscientiousness 

:=  {1-10} 

Table  3. 2. 1-8:  Example  Rule  Using  a  Personality  Factor  to 

Derive  User  Affective  State 

IF  (emotional  stability  =  low  1  med 
THEN  (anxiety  weight  factor  = 

1  high) 

101510)  AND  (mood  weight  factor  =  015110) 

Individual  history  factors  represent  specific  events  from  the  user's  history  that  may  influence 
the  current  affective  state.  Given  the  importance  of  an  individualized  approach  to  affective  and 
belief  state  assessment,  these  factors  are  among  the  most  critical,  particularly  so  since 
personality  research  indicates  that  the  most  reliable  predictor  of  future  behavior  is  past  behavior. 
A  variety  of  individual  history  factors  exist  and  must  be  selected  for  the  specific  task  context. 
Accordingly,  the  factors  used  during  the  Phase  I  ABAIS  are  customized  for  the  demonstration 
task  and  were  obtained  primarily  through  direct  interview  knowledge  elicitation  methods  from 
domain  experts.  The  selected  individual  history  factors,  and  their  possible  values  in  the  context 
of  the  Phase  I  demonstration  task,  are  summarized  in  table  3.2. 1-9.  Examples  of  rules  using 
these  factors  to  derive  an  affective  state  are  shown  in  table  3.2.1-10. 
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Table  3.2. 1-9:  Summary  of  Individual  History  Factors 
for  User  State  Assessment 


Successful  situations  in  past  :=  {Planning  I  Detection  I  Commitment  I  Sorting  I  Engagement 

i  Egress  I  Evasion  I  Hit  I  enemy  radar  lock  -  evasion  I  enemy  radar  lock  -  hit  I 
unknown  radar  lock  -  evasion  I  unknown  radar  lock  -  hit) 

Failure  situations  in  past  :=  {Planning  I  Detection  I  Commitment  I  Sorting  I  Engagement 

I  Egress  I  Evasion  I  Fratricide  I  enemy  radar  lock  -  evasion  I  enemy  radar  lock-  hit  I 
unknown  radar  lock  -  evasion  I  unknown  radar  lock  -  fratricide  I 
repeated  unsuccessful  IFF  interrogations  followed  by  fratricide} 

Affective  reactions  to  specific  people  /  events 

:=  {(flight  member  <name>  =  positive  I  negative)  I  (unknown  NCTR  type  =  (Anxiety 

_ Weight  Factor=8)) }  _ _ _ 


Table  3.2.1-10:  Examples  Rules  Using  Individual  History  Factors 
to  Derive  User  Affective  State 

IF  (<current-situation>  member-of  <past-failures>)  THEN  (AWF  =  10)  AND  (mood  factor  =  -5) 

IF  (<current-  situation>  member-of  <past-successes>)  THEN  (AWF  =  -5)  AND  (mood  factor  =  +5) 

IF  (unknown  type  from  NCTR)  THEN  (anxiety  weight  factor  =  8) 

IF  (cleared  to  fire)  THEN  (anxiety  weight  factor  =  8) 

IF  (confidence  in  wingman  =  poor  /  neutral  / good) _ THEN  (AWF  =  5/0/ -5) _ 


Dynamic  Factors  External  events  factors  represent  more  dynamic  factors  that  can 
contribute  to  the  task  difficulty  and,  again,  influence  the  overall  affective  state.  These  include  a 
variety  of  factors  relating  to  the  state  of  the  aircraft,  task  specific  factors  such  as  the  geometry  of 
the  intercept,  any  data  appearing  on  the  radar  systems,  as  well  as  specific  and  non-specific  team 
effects.  Examples  of  these  factors,  and  their  possible  values  in  the  context  of  the  Phase  I 
demonstration  task,  are  summarized  in  table  3.2.1-11.  Examples  of  rules  using  these  factors  to 
derive  an  affective  state  are  shown  in  table  3.2.1-12. 

Physiological  data  factors  represent  specific  physiological  measures  collected  from  the  user 
during  the  course  of  the  task.  Due  to  the  high  degree  of  individual  variations  in  physiological 
signals,  as  well  as  within-individual  variations  over  time  and  habituation,  these  measures  must  be 
normalized  based  on  the  user's  baseline  responsiveness  measures,  and  baseline  measures  for  the 
task  and  the  current  day.  While  a  variety  of  measures  are  theoretically  possible,  see  discussion  on 
physiological  sensing  in  section  2.3.4,  the  most  reliable  measures  of  state  anxiety  appear  to  be 
those  related  to  arousal,  that  is:  heart  rate  and  skin  conductance  measures.  Although  these 
measures  reflect  general  arousal,  rather  than  anxiety  per  se,  it  is  assumed  that  during  crisis 
situations  in  general,  and  during  the  demonstration  scenario  sweep  task  in  particular,  this  arousal 
would  be  a  likely  indication  of  anxiety.  GSR  may  not  be  a  practical  measure  in  the  fighter  pilot 
setting,  both  due  to  intrusive  monitoring  devices  (i.e.,  finger  sensors),  and  because  the  fighter 
pilot  task  environment  may  interfere  with  the  data  collection  (i.e.,  finger  sensors  may  obstruct 
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other  activities).  During  the  Phase  I  effort  we  therefore  focused  on  heart  rate  as  the  most  reliable 
and  practical  measure  of  arousal,  using  estimates  derived  from  existing  empirical  literature  and 
interviews  with  fighter  pilots.  Examples  of  assessment  rules  using  physiological  data  are  shown 
in  table  3.2.1-13. 

Table  3.2.1-11:  Summary  of  External  Events  Factors 
for  User  State  Assessment 


Equipment  failures 

:==  {engine,  nozzle  indicator,  oil  pressure,  fuel  valve  failures,  IFF,  RWR,  NCTR,  EID} 

Intercept  geometry 

:=  {single  side  offset  1  bracket  1  wall) 

Visual  contact  w /  wingman 

:=  {good  1  poor} 

Target  distance 

:=  {not  detected  1  BVR  1 WVR} 

Target  distance  change 

:==  {closing  1  opening} 

Firing  status 

:=  {cleared  to  fire  1  not  cleared  to  fire} 

NCTR  output 

:==  {none  1  unknown  1  friendly  1  hostile} 

IFF  output 

:=  {friendly  1  no  response} 

RWR  output 

:={air-to-air  unknown  1  air-to-air  hostile  1  SAM } 

Number  of  hostile  a/c 

:==  {2  1 4  1  6  1  >  6} 

Number  of  unknown  a/c 

:=  {2  1 4  1  6 1  >  6} 

Active  radar  output 

:=  {unknown  1  hostile  1  friendly} 

Team  effects 

Non-specific  :=  {team  coherence  factors} 

Specific  :={familiar  team  members  1  unfamiliar  team  members  1  liked  team  members  I  disliked  team 

1  members} 

I 

Table  3.2.1-12:  Examples  Rules  Using  External  Events  Factors  to 

Derive  User  Affective  State 


IF  (#  hostile  targets  =  0 1 2  1 4 1  >  4  1  unknown) 

THEN  (anxiety  weight  factor  =  013161918) 

IF  (#  of  contacts  increases) 

THEN  (anxiety  weight  factor  =  6) 

IF  (hostile  target  distance  =  not  detected  1  BVR  1  WVR) 

THEN  (anxiety  weight  factor  =  01517) 

IF  (distance  change  of  hostile  targets  =  closing  1  opening) 

THEN  (anxiety  weight  factor  =  510) 

IF  (IFF  output  =  friendly  1  no  response} 

THEN  (anxiety  weight  factor  =  -101 10) 

IF  (RWR  output  =  air-to-air  unknown  1  air-to-air  hostile  1  SAM)  THEN  (anxiety  weight  factor  =  711019) 

IF  (NCTR  output  =  friendly  1  unknown  1  hostile) 

THEN  (anxiety  weight  factor  =  015110) 

IF  (#  of  equipment  failures  >  1 ) 

THEN  (anxiety  weight  factor  =  5) 

IF  (engine  failure  or  flameout) 

THEN  (anxiety  weight  factor  =  7) 

IF  (fuel  system  failure) 

THEN  (anxiety  weight  factor  =  3) 

IF  (nozzle  indicator  failure) 

THEN  (anxiety  weight  factor  =  3) 

IF  (intercept  geometry  =  wall  1  single  side  offset  1  bracket) 

THEN  (anxiety  weight  factor  =  51719) 

IF  (visual  contact  w/  winqman  =  good  1  poor) 

THEN  (anxiety  weight  factor  =  015) 

Table  3.2.1-13:  Examples  of  Physiological  Factors  for  User  State  Assessment 


IF  (heart  rate  =  normal  1  high) 

THEN  (AWF  =  0110) 

IF  (respiration  rate  =  normal  1  high) 

THEN  (AWF  =  01 10) 

IF  (blood  pressure  =  normal  1  high) 

THEN  (AWF  =  0l  10) 

IF  (skin  conductance  =  normal  1  high) 

THEN  (AWF  =  0110) 
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3.2.2  Belief/  Knowledge  Assessment 

For  the  discussion  below,  we  assume  a  working  definition  of  "belief",  where  belief  state 
represents  the  currently  active  or  preferred  set  of  knowledge  constructs,  schemata,  or  procedures, 
resulting  from  a  combination  of  training,  individual  history,  and  personality  and  cognitive  style 
differences.  For  example,  a  combination  of  pilot's  training,  recent  events,  and  affective  state 
might  predispose  him  towards  a  particular  interpretation  of  existing  ambiguous  data  (e.g., 
unknown  radar  is  hostile,  approaching  unknown  aircraft  are  friendly,  etc.),  a  pilot's  training 
might  predispose  him/her  toward  a  particular  cockpit  instrument  scanning  pattern,  and  individual 
experience  might  predispose  him  to  a  specific  set  of  expectations  regarding  the  outcome  of  a 
particular  engagement.  In  other  words,  the  current  belief  state  represents  the  currently  active 
situation  schemata  and  reflects  the  pilot's  situation  assessment  and  situation  awareness.  Belief 
assessment  in  this  context  thus  corresponds  to  what  is  generally  referred  to  as  situation 
assessment  in  the  literature;  that  is,  the  identification  of  the  most  likely  current  interpretive 
schemata  guiding  situation  interpretation  and  subsequent  action  selection. 

Given  this  definition  of  beliefs,  the  following  problems  must  be  addressed  to  identify  a  belief 
state  and  its  potential  effects  on  performance.  First,  the  possible  set  of  beliefs  relevant  for  a 
,  particular  task  context  must  be  identified;  in  other  words,  the  situation  taxonomy  for  the  task 
domain  must  be  defined.  Second,  the  factors  contributing  to  the  instantiation  of  a  particular  set 
of  beliefs  during  situation  assessment  must  be  identified;  these  can  then  be  used  to  dynamically 
assess  the  pilot's  belief  state.  Finally,  a  dynamic  assessment  must  be  performed  during  the  task 
execution  to  determine  the  individual's  most  likely  set  of  active  schemata,  that  is,  the  dominant 
belief  state  and  corresponding  situation  assessment.  These  three  problems,  and  the  corresponding 
solutions  during  this  Phase  I  effort,  are  discussed  below. 

3.2.2. 1  Identifying  the  Task  Domain  Situation  Taxonomy 

The  first  problem  requires  a  detailed  ontological  analysis  of  the  task  domain,  identifying 
critical  cues,  a  taxonomy  of  possible  situations,  and  space  of  possible  actions.  This  problem  is 
best  addressed  through  a  standard  set  of  knowledge  elicitation  (KE)  techniques  and  cognitive 
task  analysis  (CTA)  methods  (Cooke,  1995;  COADE,  1995).  A  variety  of  these  methods  exist, 
including  the  direct  methods  of  protocol  analysis,  inferential  analysis,  and  dynamic  situation 
assessment,  and  a  variety  of  indirect  techniques.  Indirect  techniques  are  particularly  appropriate 
in  this  case,  since  they  are  well-suited  for  identifying  intuitive  knowledge  that  characterizes 
individual,  possibly  idiosyncratic,  beliefs,  that  are  often  difficult  to  articulate  directly.  A  number 
of  these  techniques  exist,  including  multi-dimensional  scaling,  hierarchical  clustering,  Pathfinder 
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algorithm,  and  repertory  grid  analysis  (Hudlicka,  1996). 

During  the  Phase  I  effort  we  addressed  the  initial  domain  ontological  analysis  problem 
through  the  use  of  scenario-based,  direct  interview  knowledge  elicitation  methods  and  associated 
cognitive  and  affective  task  analysis.  The  domain  characterization  (a  partial  ontological 
analysis),  yielded  the  cues,  situations,  and  actions  summarized  in  table  3. 2.2. 1-1  and  figure 
3.2.2. 1-1.  For  a  more  extensive  elicitation  effort  required  for  a  Phase  II,  other  methods  would  be 
used,  including  repertory  grid  analysis,  which  has  been  shown  to  be  an  efficient  method  for 
eliciting  large  numbers  of  individual  knowledge  constructs  and  schema  components  (Hudlicka, 
1996;  Hudlicka  et  al.,  1999). 

Table  3. 2. 2. 1-1:  Examples  of  Salient  Cues  and  Possible  Situations 


Cues 

Datalink  -  friendly  I  unknown  I  hostile 
IFF  -  friendly  I  unknown 
NCTR  -  <aircraft  type> 

RWR  -  no  radar  contact  I  hostile  radar  contact  I  friendly  radar  contact  I  SAM  radar  contact  I 
unknown  radar  contact  I 
Active  radar  -  friendly  I  unknown  I  hostile 
Targets  -  closing  I  opening 
Cleared  to  fire  notification 
Situations 

Hostile  aircraft  closing 
Hostile  aircraft  opening 
Presumed  hostile  aircraft  closing 
Presumed  hostile  aircraft  opening 
Unknown  aircraft  closing 
Unknown  aircraft  opening 
Cleared  to  fire 

Cleared  to  fire  w/  positive  EID 
Not  cleared  to  fire 

At  maximum  firing  range  for  <weapon> 

Beyond  maximum  firing  range  for  <weapon> 

Within  firing  range  of  hostile  aircraft  -  vulnerable 
Under  attack  from  hostile  aircraft 
Attacking  hostile  aircraft 
Actions 

Fire  weapon 
Initiate  intercept 
Initiate  evasive  action 
Communicate  w /  wingman 
Communicate  w /  AWACS 

Focus  on  particular  instrument  {HUD  I  radar  I  etc.} _ _ _ 
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^Situations 


[[Hostile  aircraft  closing 
Hostile  aircraft  opening 
Presumed  hostile  aircraft  closing 
Presumed  hostile  aircraft  opening 
Unknown  aircraft  closing 
Unknown  aircraft  opening 
Cleared  to  fire 

Cleared  to  fire  w /  positive  EID 
Not  cleared  to  fire 


Figure  3. 2. 2. 1-1:  Belief  Assessment  Rules  and  Resulting  Beliefs  (i.e., 
Situations)  Instantiated  in  the  Current  Task  Context 


32.2.2  Factors  Contributing  to  a  Particular  Belief  State 

A  variety  of  factors  contribute  to  the  activated  belief  state  (see  figure  3.2.2.2-1).  Each  factor 
contributes  some  piece  of  knowledge  or  evidence  to  establish,  confirm,  or  refute  a  particular 
belief  about  the  current  situation  (e.g.,  unknown  target  is  friendly  or  hostile).  The  sum  total  of 
these  influences  then  determines  the  pilot’s  overall  belief  state,  reflecting  his/her  assessment  of 
the  current  situation.  There  is  significant  overlap  in  the  knowledge  and  rules  used  to  assess  the 
affective  state  and  those  used  to  infer  the  belief  state. 

A  critical  aspect  of  the  belief  state  assessment  is  the  inclusion  of  the  pilot's  presumed 
affective  state.  This  in  effect  allows  the  implicit  modeling  of  the  influence  of  specific  affective 
states  on  cognition  and  distinguishes  the  current  approach  to  belief  /  situation  assessment  from 
existing  situation  assessment  methods  (e.g.,  SD_PVI  (Zacharias  et  al.,  1996;  Pew  and  Mavor, 
1998)).  The  knowledge-based  belief  assessment  approach  implemented  during  Phase  I  uses 
knowledge  from  each  of  these  categories  to  derive  the  user's  belief  state.  As  with  affect 
assessment  described  above,  this  approach  in  effect  emulates  an  expert  observer,  familiar  with 
both  the  task  and  the  specific  individual. 

Again,  as  with  the  affective  state  assessment,  it  is  important  to  keep  in  mind  that  the  factors, 
their  values,  and  the  corresponding  rules,  are  specified  in  the  context  of  the  current  individual- 
task  and  can  be  changed  as  that  context  changes.  In  fact,  such  individualized  tailoring  of  the 
ABAIS  knowledge-bases  is  the  key  to  its  successful  adaptation  for  a  particular  individual,  or 
group  of  individuals  in  a  team  setting. 
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Figure  3. 2. 2. 2-1:  Sources  of  Information  Used  to  Derive  Pilot’s  Belief  State 

3.2.23  Dynamic  On-Line  Belief  Assessment 

The  process  of  dynamic  belief  assessment  is  the  final  step  of  user  belief  state  assessment. 
During  this  phase  the  current  knowledge  factors  contributing  to  the  activation  of  particular 
beliefs  are  instantiated  to  derive  the  most  likely  set  of  activated  schemata,  that  is  activated  beliefs 
reflecting  the  current  situation.  This  process  essentially  simulates,  at  the  input-output  level,  the 
pilot's  own  situation  assessment  processes.  In  the  context  of  the  current  demonstration  scenario, 
the  pilot's  belief  state  is  reflected  in  the  pilot's  assessment  of  the  current  situation  from  the 
available  salient  cues.  While  in  theory  a  large  number  of  situations  are  possible,  in  practice  the 
set  of  situations  for  a  particular  task  is  generally  limited  (e.g.,  attacking  vs.  being  attacked, 
approaching  aircraft  are  friendly  or  hostile,  etc.).  In  the  context  of  the  Phase  I  demonstration 
scenario  we  therefore  limit  the  possible  situations  to  those  identified  through  initial  knowledge 
elicitation  and  cognitive  task  analysis  (see  table  3.2.2.1-1).  Specifically,  the  primary  cues  are  the 
radar  returns  and  datalink  information  available  on  the  aircraft's  detection  systems;  that  is,  the 
active  TWS  air-to-air  radar,  NCTR,  IFF,  and  RWR.  The  critical  situations  then  consist  of  the 
identity,  distance,  and  course  of  identified  targets,  whether  or  not  the  aircraft  is  within  a  weapons 
range,  whether  or  not  the  aircraft  is  vulnerable,  and  whether  or  not  the  pilot  is  cleared  to  fire. 

The  most  common  existing  approaches  to  belief  and  situation  assessment  are  AI  knowledge- 
based  methods  using  one  of  a  variety  of  knowledge  representation  formalisms  and  associated 
inferencing  methods  (e.g.,  rule  based  reasoning,  case  based  reasoning,  Bayesian  belief  nets,  or 
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various  hybrid  combinations  thereof)  (Pew  and  Mavor,  1998). 

Under  the  Phase  I  effort,  we  selected  a  rule-based  approach  for  belief  assessment.  As  was 
the  case  with  affective  state  assessment  discussed  above,  the  knowledge  used  to  dynamically 
derive  the  pilot’s  belief  state  was  encoded  in  terms  of  production  rules.  The  actual  belief  state 
was  then  derived  from  a  combination  of  static,  a  priori  information  about  the  pilot  and  the  task, 
and  from  dynamic  data  reflecting  the  changing  task  environment  and  pilot  state,  including  the 
pilot’s  affective  state.  The  most  critical  categories  of  factors  were:  1)  external  events  (i.e.,  cues); 
2)  individual  history;  and  3)  current  affective  state.  These  are  described  in  detail  below. 


External  events  External  events  represent  the  dynamic  task  environment  and  include  both 
events  on  the  aircraft  itself  (e.g.,  instrument  failures),  and  events  in  the  task  environment  (e.g., 
behavior  of  other  aircraft,  both  friendly  and  enemy;  incoming  sensor  radar  data).  Examples  of 
external  events  are  listed  in  table  3.2.2.1-1  above  and  in  figure  3.2.2.3-1  below. 


|||1  Cues 

m\ 

IFF  -  friendly  |  unknown 
NCTR  -  -^aircraft  1ype=- 

RWR  -  no  radar  contact  |  hostile  radar  contact  | 
friendly  radar  contact  |  SAM  radar  contact  | 
unknown  radar  contact 
Active  radar-  friendly  |  unknown  |  hostile 
Targets  -  closing  |  opening 
Cleared  to  fire  notification 


Figure  3. 2.2. 3-1:  Examples  of  Cues  Used  as  Input 
for  Belief  State  Assessment 


Individual  history  combines  the  training  and  skill  factors  with  specific  experiences  that 
influence  the  pilot's  situation  assessment  and  decision-making.  In  other  words,  specific 
successful  or  unsuccessful  experiences  that  tend  to  predispose  the  pilot  towards  or  against  certain 
situations  and  maneuvers.  For  example,  in  the  current  demonstration  scenario,  occurrence  of 
specific  recent  situations  may  bias  the  interpretation  of  current  data;  i.e.,  if  the  pilot  has  recently 
experienced  a  situation  where  a  number  of  unsuccessful  IFF  interrogations  were  followed  by  a 
final  identification  of  that  aircraft  as  hostile,  s/he  may  be  predisposed  to  conclude  that  if  an 
aircraft  does  not  respond  to  IFF  interrogations  it  is  in  fact  hostile. 

Current  affective  state  The  pilot's  affective  state  plays  a  critical  role  in  his/her  situation 
assessment.  By  taking  into  account  the  current  affective  state,  the  ABAIS  User  Assessment 
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module  in  effect  implicitly  models  the  potential  biasing  influences  of  the  different  affective 
states  and  provides  a  structure  which  allows  the  explicit  representation  of  the  positive  feedback 
between  cognition  and  affect  that  is  often  seen  in  crisis  situations.  In  other  words,  increased 
anxiety  contributes  to  a  particular  situation  assessment  (e.g.,  aircraft  is  being  attacked  by  hostile 
aircraft),  which  then  limits  the  processing  of  data  that  could  give  rise  to  alternative 
interpretations  and  further  increases  the  anxiety  level.  Examples  of  rules  for  belief  assessment 
are  shown  in  figure  3.2.23-2  and  table  3. 2.2.3- 1.  These  rules  map  the  combinations  of  cues 
representing  external  events,  individual  history,  and  affective  state  onto  the  set  of  possible 
situations,  which  are  summarized  in  table  3.2.23-2. 


[F  (unknown  air-to-air  radar  lock) 
AND  (repeated  no  response  IFF) 
THEN  (target  is  hostile) 

AND  (aircraft  is  under  attack) 

IF  (uknown  targets  closing) 

AND  (anxiety  level  high) 

THEN  (targets  =  hostile) 

IF  (targets  unknown) 

AND  (anxiety  level  =  high) 

THEN  (targets  hostile) 

IF  (unknown  type  from  NCTR) 

AND  (anxiety  level  =  high) 

THEN  (targets  hostile) 

IF  (air-to-air  radar  lock  on  aircraft) 


Figure  3.2. 2.3-2:  Examples  of  Belief  Assessment  Rules 
Table  3. 2. 2. 3-1:  Examples  of  Rules  Deriving  Pilot's  Belief  State 


IF  (unknown  contacts  closing)  AND  (anxiety  level  >  low) 

THEN  (contacts  =  hostile) 

IF  (unknown  contacts  closing)  AND  (anxiety  level  =  low) 

THEN  (contacts  =  unknown) 

IF  (cleared  to  fire)  AND  (unknown  contacts  closing) 

THEN  (contacts  =  hostile) 

IF  (unknown  type  from  NCTR)  AND  (anxiety  level  =  >  low) 

THEN  (contact  =  hostile) 

IF  (unknown  type  from  NCTR) 

THEN  (contact  =  hostile) 

IF  (unknown  air-to-air  radar  lock)  AND  (repeated  no  response  IFF) 

THEN  (target  is  hostile)  AND  (aircraft  is  under  attack) 

IF  (unknown  targets  closing)  AND  (anxiety  level  high) 

THEN  (targets  =  hostile) 

IF  (target  =  unknown)  AND  (anxiety  level  =  high) 

THEN  (target  =  hostile) 

IF  (air-to-air  radar  lock  on  aircraft)  AND  (anxiety  level  =  high) 

THEN  (radar  lock  =  hostile) 

IF  (unknown  air-to-air  radar  lock  on  aircraft)  AND  (hostile  SAM  lock  on  aircraft) 

THEN  (aircraft  is  under  attack) 
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Table  3.2.2.3-2:  Examples  of  Belief  Assessment  Output:  Possible  Situations 

for  each  ~  ~  - 

target  on  active  radar  =  {hostile  I  unknown  I  friendly) 

for  each 

radar  warning  signal  =  {hostile  I  unknown  I  friendly) 

cleared  to  fire  =  {yes  I  no) 

under  attack  =  {yes  I  no  I  unknown) 

at  max  firing  range  for  <weapon>  =  {yes  I  no) 

vulnerable _  =  {yes  I  no  I  unknown) _ 

3.3  Impact  Prediction  Module 


The  overall  functionality  of  Impact  Prediction  Module  is  summarized  in  figure  33-1.  The 
Impact  Prediction  Module  receives  as  input  the  identified  affective  and  belief  states  and 


Figure  3.3-1:  Functionality  of  the  Impact  Prediction  Module 


The  goal  of  the  impact  prediction  module  is  to  predict  the  influence  of  a  particular  affective 
state  (e.g.,  high  anxiety)  or  belief  state  (e.g.,  "aircraft  under  attack",  "hostile  aircraft 
approaching",  etc.)  on  task  performance.  Impact  prediction  thus  represents  an  essential 
component  of  the  overall  adaptation  strategy.  Impact  prediction  process  implemented  under 
Phase  I  uses  rule-based  reasoning  (RBR)  and  takes  place  in  two  stages.  First,  the  generic  effects 
of  the  identified  affective  state  are  identified,  using  a  knowledge-base  that  encodes  empirical 
evidence  about  the  influence  of  specific  affective  states  on  cognition  and  performance.  Next, 
these  generic  effects  are  instantiated  in  the  context  of  the  current  task  to  identify  task-specific 
effects,  in  terms  of  relevant  domain  entities  and  procedures  (e.g.,  task  prioritization,  threat 
assessment).  The  knowledge  encoded  in  these  rules  is  derived  from  a  detailed  cognitive  / 
affective  task  analysis,  which  predicts  the  effects  of  different  affective  states  on  performance  in 
the  current  task  context.  Such  a  task  analysis  is  a  critical  component  of  building  the  impact 
prediction  knowledge  base,  since  the  state-of-the-art  of  theoretical  understanding  and  empirical 
research  in  personality  and  emotion  do  not  allow  accurate  prediction  of  these  influences  in  a 
generic,  domain-independent  manner.  The  separation  of  the  generic  and  specific  knowledge 
enhances  modularity  and  simplifies  knowledge-based  adjustments. 

3.3.1  Effects  of  Affective  States  and  Personality  Traits  on  Performance 

Generic  Effects  Primary  source  of  evidence  for  the  generic  (i.e.,  domain  independent) 
impacts  of  affective  states  on  performance  was  empirical  literature  in  experimental  and  cognitive 
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psychology.  Several  affective  states  and  personality  traits  have  been  studied  extensively  in 
laboratory  settings.  Empirical  findings  consistently  indicate  that  anxiety  affects  attention  and 
perception;  mood  affects  recall,  prediction  of  future  outcomes,  and  a  sense  of  locus  of  control, 
and  obsessiveness  affects  decision  making  speed  and  recall  of  recent  events.  Table  3.3. 1-1 
summarizes  findings  from  experimental  psychology  regarding  the  generic  effects  of  selected 
affective  states  and  personality  traits  on  performance.  The  remainder  of  this  section  discusses  in 
more  detail  both  the  generic  effects,  and  their  specific  instantiations  in  the  context  of  the 
demonstration  task. 

Table  3.3.1-1:  Summary  of  Generic  Effects  of  Selected  Affective  States  and 

Personality  Traits  on  Performance 

Anxiety  and  Attention  (Williams  et  a!.,  1997;  Mineka  and  Sutton,  1992;  Eysenck,  1997) 

Narrowing  of  attentional  focus 
Predisposing  towards  detection  of  threatening  stimuli 
Mood  and  Memory  (Bower,  1981;  Blaney,  1986) 

Mood-congruent  memory  phenomenon  -  particular  affective  state  induces  recall  of  similarly  valenced  material 
Obsessiveness  and  Performance  (Broadbent  et  al.,  1986;  Persons  &  Foa,  1984;  Sher  et  al.,  1989) 

Delayed  decision-making 

Reduced  ability  to  recall  recent  activities 

Reduced  confidence  in  ability  to  distinguish  among  actual  and  imagined  actions  and  events 
Narrow  conceptual  categories 
Affect  and  Judgment  &  Perception  (Isen,  1993) 

Depression  lowers  estimates  of  degree  of  control 

Anxiety  predisposes  towards  interpretation  of  ambiguous  stimuli  as  threatening _ 


Specific  Effects  Although  there  appears  to  be  an  increasing  interest  in  linking  specific 
personality  traits  and  affective  states  with  performance,  there  are  surprisingly  few  systematic 
empirical  studies  investigating  these  effects,  at  a  level  of  detail  necessary  for  a  knowledge-based 
adaptation  system  such  as  ABAIS.  In  other  words,  while  studies  have  been  conducted  that 
attempt  to  correlate  training  success  or  career  performance  with  broad  personality  measures  (e.g., 
“Big  5’),  little  work  has  been  done  in  identifying  specific  performance  biases  induced  by 
transient  states  of  anxiety  and  other  affects,  or  by  systematic  biases  due  to  dominant  personality 
traits.  Due  to  the  paucity  of  systematic  empirical  studies  in  this  area,  the  primary  source  of 
knowledge  for  the  identification  of  specific  effects  under  Phase  I  were  SME’s  (i.e.,  experienced 
fighter  pilots).  Knowledge  elicitation  interviews  with  multiple  SME’s  provided  information 
about  the  likely  effects  of  a  variety  of  affective  states  and  personality  traits  on  pilot  performance. 
The  results  of  these  interviews  are  summarized  below. 
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Anxiety  -  Attention  and  Perception:  The  generic  effects  of  anxiety  on  attention 
include  narrowing  of  attentional  focus,  difficulty  focusing  attention  (i.e.,  inability  to  select  an 
action  and  consequent  delayed  reaction  time),  and  increased  attention  to  threatening  stimuli. 
This  narrowing  of  attention  may  result  in  task  neglect  for  other  critical  tasks,  and  a  failure  to 
detect  other  relevant  cues. 

Given  this  generic  knowledge,  a  detailed  cognitive  /  affective  task  analysis  is  then  used  to 
predict  situations  where  these  biases  may  influence  performance.  In  other  words,  the  cognitive  / 
affective  task  analysis  identifies  task  segments  where  parallel  signals  may  occur  (e.g.,  two 
signals  on  radar  from  two  different  sources,  radar  and  engine  instruments,  etc.)  and  identify 
points  where  parallel  tasks  take  place.  These  then  allow  predictions  as  to  which  of  these  tasks  is 
likely  to  be  neglected  during  a  state  of  increased  anxiety  (e.g.,  pilot  is  more  likely  to  pay 
attention  to  radar  signals  than  engine  instruments  or  radio). 

The  generic  effect  of  anxiety  on  perception  is  to  bias  perception  towards  interpretation  of 
ambiguous  signals  as  threats.  Since  the  primary  perceptual  bias  is  the  misinterpretation  of  cues, 
the  cognitive  /  affective  task  analysis  required  to  identify  specific  possible  instances  must  focus 
on  potential  instances  of  ambiguous  cues,  and  cues  likely  to  be  misinterpreted  as  threats,  and 
identify  from  these  candidates  for  likely  misinterpretation  as  threats. 

This  type  of  task  analysis,  based  on  knowledge  elicitation  data  from  SME’s,  identified  a 
number  of  specific  effects  of  the  generic  anxiety-induced  biases  discussed  above.  These  include: 

•  Focussing  on  target  on  radar  display  and  failing  to  notice  incoming  communication  from 

other  sources  (e.g.,  radio  voice  communication  from  wingman,  AWACS,  etc.). 

•  Focussing  on  target  information  on  HUD  and  failing  to  notice  new  information  on  radar. 

•  Focussing  on  “centering  the  dot”  and  failing  to  notice  a  change  in  icon  indicating  a  change  in 

target  identification  (from  hostile  or  unknown  to  friendly). 

•  Interpreting  any  of  the  various  ambiguous  signals  on  radar  as  threats.  This  includes 

ambiguous  IFF  signals,  RWR  signals  that  do  not  indicate  whether  the  aircraft  is  being 
'painted'  by  a  hostile  or  friendly  radar;  NCTR  which  provides  information  about  the  type  of 
aircraft  but  not  necessarily  whether  it  is  hostile  or  friendly;  unknown  NCTR  signal.3 

•  Focussing  on  information  on  radar  and  failing  to  notice  cockpit  warnings  of  aircraft  system 

malfunctions. 

Examples  of  specific  rules  constructed  from  this  knowledge  are  shown  in  table  3.3. 1-2. 


3For  example,  a  MIG  fighter  aircraft  could  be  owned  by  a  friendly  ally  or  an  adversary.  Thus  an  indication  that 
an  aircraft  is  a  MIG  may  not  indicate  whether  it  is  friendly  or  hostile. 
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Table  3.3. 1-2:  Example  Rules  Predicting  the  Effect 
of  Anxiety  on  Performance 

Anxiety-Induced  Narrowing  of  Attention 

IF  (multiple  targets  on  radar)  THEN  (focused  on  signals  representing  unknowns  or  threats) 

IF  (multiple  unknown  /  threats  targets  on  radar)  THEN  (focused  on  nearest  /  fastest-approaching  targets) 

IF  (data  arriving  on  multiple  cockpit  instrument)  THEN  (attention  focused  on  radar  or  HUD) 

IF  (data  arriving  on  HUD  and  radar)  THEN  (attention  focused  on  HUD) 

Anxiety-Induced  Misinterpretation  of  Cues 

IF  (unknown  or  ambiguous  targets  on  radar  I  NCTR  exist)  THEN  (assume  targets  are  hostile) 

IF  (unknown  air-to-air  radar  lock  on  RWR)  THEN  (assume  under  attack) 

IF  (no  reply  from  IFF)  THEN  (assume  targets  are  hostile) _ _ _ 

Mood  -  Memory  and  Judgment:  The  generic  effects  of  mood  on  memory  and 
judgment  include  the  mood-congruent  recall  phenomenon  and  mood-congruent  outcome 
prediction.  In  mood  congruent  recall  the  individual's  mood  determines  the  type  of  information 
recalled;  in  other  words,  positive  mood  favoring  the  recall  of  positively-valenced  information 
(e.g.,  successful  past  situation),  and  negative  moods  favoring  the  recall  of  negatively-valenced 
information  (e.g.,  failures  from  the  past).  In  mood-congruent  prediction  of  future  outcomes, 
positive  moods  favor  positive  estimates  of  outcomes  and  negative  mood  favor  negative  estimates 
of  outcomes.  The  exact  content  of  the  recalled  events  is  of  course  a  function  of  the  individual- 
task  context. 

In  the  context  of  the  sweep  task  scenario,  these  generic  effects  of  mood  on  memory  and 
judgement  may  result  in  the  following  specific  effects: 

•  A  pilot  in  a  negative  mood  may  avoid  a  particular  situation,  action,  or  maneuver  because  it 

reminds  him/her  of  past  failures  and  because  s/he  anticipates  a  negative  outcome. 

•  Conversely,  a  pilot  in  a  positive  mood  may  initiate  a  particular  action  or  maneuver  because  it 

reminds  him/her  of  successful  situations  from  the  past  and  because  s/he  anticipates 
successful  outcome. 

An  example  of  the  former  situation  is  a  lead  pilot  who  has  recently  experienced  a  situation 
where  he  became  separated  from  his  wingman  while  attempting  to  change  formation,  lost  sight 
of  his  wingman,  and  subsequently  the  wingman  was  shot  down.  There  is  an  increased  likelihood 
that  this  pilot  will  avoid  similar  maneuvers  in  the  future,  particularly  when  in  a  negative  mood, 
which  increases  the  likelihood  of  recalling  this  "negative"  situation  from  the  past. 

To  successfully  instantiate  these  effects,  detailed  description  of  the  pilot’s  individual  history 
is  required,  where  specific  situations  from  the  past  are  linked  with  their  associated  affective 
valence  (i.e.,  positive  or  negative). 
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Obsessiveness  -  Decision-Making:  The  generic  effects  of  obsessiveness  on  decision 
making  include  delayed  decision-making,  reduced  ability  to  recall  recent  events,  narrow 
conceptual  categories,  and  inability  to  adapt  to  new  and  unanticipated  situations. 

In  the  context  of  fighter  pilot  tasks,  these  generic  effects  may  produce  what  is  known  as  a 
"checklist"  mentality,  which  may,  in  some  situations,  take  up  valuable  time  and  lead  to  delayed 
decision  making.  A  cognitive  /  affective  task  analysis  of  the  sweep  task  indicates  that  the 
generic  effects  of  obsessiveness  on  decision  making  described  above  may  result  in  any  of  the 
following  specific  effects: 

•  Failing  to  recall  recent  events  may  result  in  repetition  of  procedure  steps  or  assumption  that 

the  steps  have  already  been  performed 

•  Repeated  IFF  interrogations 

•  Rechecking  the  weapon  switches  (e.g.,  master  arm  switch,  weapon  selection  switch,  radar 

mode) 

•  Repeating  verbatim  messages  from  AWACS  rather  than  simply  acknowledging  them  by 

saying  "Roger" 

•  Repeating  messages  received  from  AWACS  to  wingman,  even  though  wingman  also  receives 

all  AWACS  communication. 

•  Verifying  wingman's  sort  even  though  wingman  in  principle  remembers  this  from  pre-flight 

briefings  and  executes  this  correctly  during  flight. 

These  "checking"  behaviors  may  then  delay  decisions  such  as  committing,  initiating  a 
maneuver  such  as  changing  formation,  firing  a  weapon,  or  turning  to  commit. 

Impulsivity  and  Performance:  Empirical  evidence  indicates  that  high-impulsives 
seek  out  conditions  of  high  arousal  and  tend  to  perform  better  under  conditions  of  high  arousal. 
In  contrast,  low  impulsivity  tends  to  be  associated  with  preference  for  familiar  situations  and 
possible  a  lower  degree  of  adaptability. 

Contrary  to  what  one  might  intuitively  expect,  impulsivity  does  not  necessarily  result  in 
riskier  behavior  per  se,  but  rather  in  faster  decisions  made  on  the  basis  of  less  evidence,  which 
may  or  may  not  be  'risky'. 

In  the  context  of  fighter  pilot  tasks,  these  generic  effects  of  high  impulsivity  may  result  in  any 
of  the  following: 

•  Faster,  'snap',  decisions  and  judgements  made  on  the  basis  of  less  information 

•  Increased  likelihood  of  identifying  an  ambiguous  return  as  hostile 

•  Increased  likelihood  in  performing  more  aggressive  maneuvers. 
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3.4  Strategy  Selection  Module 

The  overall  functionality  of  the  Strategy  Selection  Module  is  summarized  in  figure  3.4-1. 
The  Strategy  Selection  Module  receives  as  input  the  predicted  specific  effects  of  the  affective  and 
belief  states  and  selects  a  compensatory  strategy  to  counteract  performance  biases  resulting  from 
these  effects. 


Figure  3.4-1:  Functionality  of  the  Strategy  Selection  Module 

Strategy  selection  is  accomplished  by  rule-based  reasoning.  The  knowledge-base  contains 
mappings  between  specific  performance  biases  identified  by  the  Impact  Prediction  Module  (e.g., 
task  neglect,  threat-estimation  bias,  failure-estimation  bias,  etc.)  and  associated  compensatory 
strategies  (e.g.,  present  reminders  of  neglected  tasks,  present  broader  evidence  to  counteract 
threat-estimation  bias,  present  contrary  evidence  to  counteract  failure-driven  confirmation  bias, 
etc.).  As  was  the  case  with  impact  prediction,  the  strategy  selection  module  relies  on  a  detailed 
analysis  of  the  task  context  that  identifies  specific  strategies  available  to  counteract  the  possible 
biases.  This  analysis  then  allows  the  construction  of  the  strategy  selection  knowledge  bases. 
Again,  the  process  is  divided  into  generic  and  task  specific  components,  to  facilitate  KB  editing 
and  modification.  Generic  strategies  map  the  generic  performance  bias  (e.g.,  task  neglect)  into  a 
generic  compensatoiy  strategy  (e.g.,  present  reminders  of  neglected  tasks).  Table  3.4-1  lists 
examples  of  these  generic  strategy  rules. 

The  generic  strategies  are  then  instantiated  in  the  specific  strategy  rules,  based  on  knowledge 
about  the  task  context  and  detailed  cognitive  /  affective  task  analysis.  Examples  of  specific  rules 
for  compensatory  strategy  selection  are  shown  in  table  3.4-2. 
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Table  3.4-1:  Examples  of  Generic  Rules  for 
Compensatory  Strategy  Selection 
Anxiety  effects  ~  '  ~  ~~  “  ~  '  " 

IF  (<task>  importance  =  high)  AND  (pilot’s  assessment  of  <task>  importance  =  low) 

THEN  (present  reminders  for  <task>)  AND  (direct  attention  to  neglected  instruments  /  data) 
IF  (threat  estimation  bias  =  high) 

THEN  (collect  all  available  evidence  regarding  radar  signal  identity) 

IF  (confirmation  bias  =  high) 

THEN  (obtain  any  contradictory  evidence)  AND  (enhance  display  of  evidence) 

Obsessiveness  effects 

IF  (obsessiveness  =  high) 

THEN  (remind  of  consequences  of  delayed  decisions) 

(remind  that  no  data  exist  that  will  provide  additional  information) 

(remind  of  most  recent  tasks  accomplished  -  present  explicit  checklists) 

_ (display  task  timeline  and  current  position  within  timeline) _ 


Table  3.4-2:  Examples  of  Specific  Rules  for 
Compensatory  Strategy  Selection 


Anxiety  effects 

IF  (recent  change  in  radar  target  status)  THEN 
(emphasize  change  in  status  of  return) 

IF  (attention  focus  =  HUD)  AND  (incoming  radar  data)  THEN 
(redirect  focus  to  radar) 

IF  (attention  focus  =  radar)  AND  (incoming  radio  call)  THEN 
(redirect  focus  to  radio) 

IF  (attention  focus  =  non-radar  instruments)  AND  (incoming  radar  data)  THEN 
(redirect  focus  to  radar) 

IF  (likelihood  of  task  neglect  for  <instrument>  =  high)  &  (has-critical-info?  <instrument>) 
THEN  (emphasize  <instrument>  visibility) 

IF  (target  =  unknown)  AND  (target  belief  =  hostile)  THEN 
(emphasize  unknown  status)  AND  (collect  more  data) 


Aggressiveness  effects 

IF  (likelihood  of  premature  attack  =  high)  THEN 
(display  all  available  info  about  enemy  a/c)  AND 
(enhance  display  of  enemy  a/c  info) 

Obsessivness  effects 

IF  (likelihood  of  delayed  attack  =  high)  THEN 
(display  all  available  info  about  enemy  a/c)  AND 
(display  likelihood  of  attack  by  enemy  a/c)  AND 
(display  envelope  of  vulnerability  around  own  aircraft)  AND 
(display  reminders  for  attack  tasks! _ 


3.5  GUI  /  DSS  Adaptation  Module 

The  GUI/  DSS  Adaptation  Module  performs  the  final  step  of  the  adaptive  methodology  by 
implementing  the  selected  compensatory  strategy  in  terms  of  specific  GUI  modifications.  The 


70 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


overall  functionality  of  this  module  is  summarized  in  figure  3.5-1.  A  rule-based  approach  is  used 
to  encode  the  knowledge  required  to  map  the  specific  compensatory  strategies  onto  the  necessary 
GUI  /  DSS  adaptations. 


Figure  3.5-1:  Functionality  of  the  GUI/DSS  Adaptation  Module 


Modes  of  Adaptation  In  general,  two  broad  categories  of  adaptation  are  possible: 

•  Content-Based  adaptations,  which  provide  additional  information,  and 

•  Format-Based  adaptations,  which  modify  the  format  of  available  information. 

Content-based  adaptation  involves  the  collection  and  display  of  additional  data  or 
knowledge  to  prevent  a  particular  performance  bias.  For  example,  providing  additional  data 
about  an  ambiguous  radar  signal  may  help  prevent  an  anxiety-induced  confirmation  bias  to 
identify  the  signal  as  a  threat.  In  the  Phase  I  ABAIS  demonstration  scenario,  this  type  of 
adaptation  was  demonstrated  in  the  context  of  demonstration  scenario  #2,  focusing  on 
adaptations  due  to  obsessive  behavior.  In  this  case,  a  compensatory  strategy  was  selected  to 
counteract  the  pilot’s  obsessive  “checking”  behavior  which  was  delaying  his  decision  to  fire.  The 
content-based  adaptation  strategy  first  identified  additional  relevant  information  (i.e.,  the  fact 
that  the  friendly  aircraft  was  now  within  the  hostile  aircraft’s  missile  range),  and  then  displayed 
this  information  in  a  prominent  manner,  to  warn  the  pilot  of  consequences  of  further  delaying  his 
decision  to  fire  (see  figure  3.5-2).  The  red  triangle  overlaid  on  the  radar  display  indicates  the 
range  of  the  hostile  missile.  The  fact  that  the  aircraft  is  within  this  triangle  indicates  that  the  pilot 
is  vulnerable. 


Format-based  adaptation  involves  the  presentation  of  existing  data  in  an  alternative  format, 
to  enhance  visibility  and  /  or  to  draw  attention  to  neglected  displays,  and,  in  general,  to  facilitate 
fast  detection,  recognition,  and  assimilation,  thereby  improving  situation  awareness.  In  the  Phase 
I  ABAIS  demonstration  scenario,  this  type  of  adaptation  was  demonstrated  in  the  context  of 
demonstration  scenario  #1,  focusing  on  adaptations  due  to  high  levels  of  anxiety.  In  this 
scenario,  a  compensatory  strategy  was  selected  to  counteract  the  pilot’s  narrowed  attention  and 
inaccurate  situation  assessment.  Namely,  his  failure  to  note  a  recent  change  in  status  of  a  contact 
from  presumed  unknown  to  friendly,  and  his  consequent  failure  to  cease  preparation  for  firing  a 
missile.  The  format-based  adaptation  first  identified  possible  strategies  for  enhancing  the 
neglected  information  (i.e.,  highlighting  the  radar  contact  whose  status  has  recently  changed), 
and  then  implemented  these  strategies  in  terms  of  specific  modifications  of  the  pilot’s  displays 
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(see  figure  3.5-2).  Namely,  highlighting  the  contact  in  question  on  the  pilot’s  HUD  (blue  box) 
and  highlighting  the  status  information  on  the  radar  display  (blinking  blue  text  message  box  in 
lower  left  comer  of  radar  display). 


Scenario  112  GUI  Adaptation  H0*IE3 


Figure  3.5-2:  Example  of  a  Content-Based  Adaptation  Aiming  to 
Compensate  for  an  Obsessive  Delaying  of  the  Decision  to  Fire 

The  arrow  on  right  indicates  pilot’s  current  focus.  The  red  triangle  indicates 
the  area  of  vulnerability  from  hostile  aircraft  missile.  The  blue,  blinking 
notification  string  further  emphasizes  the  friendly ’s  vulnerability. 
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Figure  3.5-3:  Example  of  a  Format-Based  Adaptation  Aiming  to  Compensate 
for  an  Anxiety-Induced  Narrowing  of  Attention  and  Perception  Bias 

The  arrow  on  right  indicates  pilot’s  current  focus.  The  blinking,  enlarged,  blue 
contact  icon  on  the  HUD  indicates  a  change  in  status.  The  blinking,  blue 
contact  icon  on  the  radar  indicates  a  change  in  status,  with  details  provided  in 
the  text  box  in  lower  left  corner  of  the  radar  display. 

Level  of  Adaptation  Regardless  of  the  method  chosen,  adaptation  eventually  results  in  a 
modification  of  specific  user  interface  (UI)  attributes.  These  include  changes  in  overall  format, 
different  choice  of  icons,  or  changes  in  UI  elements,  including  color,  location,  size,  orientation, 
modality,  motion  of  stimulus,  motion  on  periphery  to  re-direct  attention  by  pre-attentive 
processes.  Adaptation  can  thus  take  place  at  any  of  the  four  levels  below: 

•  Icon  level 

•  Display  level 

•  Notification  level 
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Each  of  these  levels  affords  different  alternatives  and  is  more  or  less  suitable  for  a  given 
situation,  depending  on  the  task,  task  context,  and  the  individual.  The  different  levels  of 
adaptation  are  discussed  below. 

Icon  level  adaptation  involves  modifications  to  the  individual  icons  within  a  particular 
display.  For  example,  changing  the  size  or  appearance  of  an  icon  in  some  way  to  enhance  its 
visibility  (e.g.,  highlighting,  changing  its  location  within  a  display,  changing  color  or  size,  etc.). 
It  can  also  include  modifying  the  icon  proper,  for  example,  making  it  less  abstract  or  more 
graphical,  to  eliminate  the  presentation  of  complex  alpha-numeric  data  and  associated  perceptual 
and  cognitive  load  when  the  user’s  resources  are  limited  due  to  increased  levels  of  anxiety. 

Display  level  adaptation  involves  modifications  of  the  display  as  a  whole,  rather  than  the 
individual  icons  within  the  display.  Examples  of  display  level  adaptations  are  changing  the  size 
or  location  of  a  selected  display  (e.g.,  moving  a  critical  display  to  a  central  location  of  the  overall 
UI),  changing  the  appearance  of  a  display  (e.g.,  range  setting  on  radar),  or  changing  the  contents 
of  a  display  (e.g.,  decluttering  a  display). 

Notification  level  adaptation  involves  inserting  new,  or  modifying  existing,  alarms  and  alert 
notifications.  Examples  of  notification  level  adaptations  include  adding  a  notification  string 
regarding  desired  focus  of  attention  (see  for  example  “RADAR”  on  the  HUD  display  in  figure 
3.5-3  or  “VULNERABLE”  string  on  the  radar  display  in  figure  3.5-2),  or  adding  an  icon  to  a 
display  to  represent  new  information  (see  “vulnerability”  triangle  in  figure  3.5-2). 

Finally,  UI  level  adaptation  involves  global  changes  to  the  user  interface  as  a  whole  or 
insertion  of  display  elements  designed  to  focus  attention  on  particular  areas  of  the  overall  UI. 
Examples  of  UI  level  adaptations  include  a  reconfiguration  of  the  entire  set  of  instruments  on  the 
UI  to  reflect  a  different  system  mode,  increasing  the  redundancy  of  warnings  (e.g.,  adding  an 
auditory  warning  to  a  visual  one,  etc.),  or  the  insertion  of  attention-capturing  and  attention¬ 
directing  elements  designed  to  direct  the  user's  attention  to  a  particular  icon  or  display. 

Individualized  Adaptation  All  of  these  strategies  are  further  enhanced  and  customized 
by  taking  into  account  the  user’s  (e.g.,  the  pilot’s)  display  and  modality  preferences  and 
implementing  the  suggested  GUI  adaptation  accordingly.  This  element  is  critical  to  any  adaptive 
approach,  due  to  the  large  individual  differences  that  exist  in  human  information  processing  and 
decision-making.  ABAIS  allows  the  specification  of  multiple  pilot  information  preference 
profiles,  which  indicate  the  individual  preferences  for  presenting  information.  For  example, 
knowing  that  a  particular  pilot  has  a  high-sensitivity  to  auditory  signals,  ABAIS  suggests  that 
auditory  warnings  be  used  to  capture  attention.  In  another  case,  a  particular  pilot  might  have  a 
preferred  means  of  capturing  attention,  for  example,  by  motion  on  the  periphery  towards  the 
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desired  area.  Examples  of  specific  pilot  profile  information  from  the  Phase  I  demonstration  are 
shown  in  figure  3.5-4  and  table  3.5-1. 


Under  Phase  I  this  included  preferred  modality  for  alarms  and  notifications,  preferred  modes 
of  attention  capture.  The  same  mechanism  can  be  extended  to  include  preferences  for  particular 
icons,  particular  displays,  and  display  configurations. 


H|GLH  Adaptation  Preferences 


Color 


Figure  3.5-4:  Pilot  Information  Preference  Profile  Dialog  Box 


Table  3.5-1:  Pilot  Information  Preference  Profile 


Enhance  visibility  by  changing: 

color  1  size  1  blinking 

Preferred  color  for  alarms: 

red  solid  1  red  outline 

Alarm  notification: 

visual  1  auditory 

Attention  capture  preference: 

movement  at  periphery  1  shift  display  to  foveal  region  1 
enhance  visibility  of  icon  /  display  1  display  arrow 
pointing  to  desired  icon  /  display  1  superimpose  blinking 
icon  /  displav  over  foveal  area 

The  GUI  /  DSS  adaptation  strategies  are  expressed  in  abstract  terms,  and  are  instantiated 
within  the  particular  user-task  context,  taking  into  consideration  the  user  preference  profiles.  The 
GUI/  DSS  Adaptation  Module  performs  three  sequential  functions: 

•  Identifies  additional  information  required  based  on  selected  compensatory  strategy 

•  Selects  best  information  presentation  format 

•  Applies  individual  information  presentation  preferences  and  capabilities  (e.g.,  modality 

preference,  color  blindness,  etc.). 

Examples  of  specific  adaptation  rules  from  the  Phase  I  demonstration  are  shown  in  figure 
3.5-5. 
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g|GUI  Adaptation 


IF  (goal  =  increase  visibility  of  radar  icon) 

THEN(enhance  icon  visibility  according  to  subject  profile  preference) 

IF  (goal  =  redirect  attention  to  instrument) 

THEN(implement  attention  capture  preference  according  to  subject  profile 
IF  (goal  =  enhance  <instrument>  visibility) 

AND  (color  preference  =  <color>) 

THEN(highlight  instrument  in  <color>) 

jlF  (goal  =  ensure  <instrumen  t>  visibility)  ' 

THEN(apply  attention  capture  method  to  ^instruments 


Figure  3.5*5:  Examples  of  ABAIS  GUI  /  DSS  Adaptation  Rules 


3.6  ABAIS  Graphical  User  Interface  (GUI) 

The  ABAIS  graphical  user  interface  (GUI)  consists  of  two  sets  of  displays:  those 
representing  the  cockpit  instruments  (the  pilot’s  GUI),  and  those  used  by  the  analyst  to  specify 
the  background  information  required  by  ABAIS,  to  define  the  run-time  parameters,  and  to 
monitor  system  performance  during  a  simulation  -  adaptation  run. 

3.6.1  Pilot’s  GUI 

The  pilot’s  GUI  (figure  3.6.1-1)  consists  of  four  displays,  corresponding  to  the  heads-up- 
display  (HUD)  (upper  portion  of  the  display),  a  window  showing  current  incoming 
communication  (middle  pink  window),  an  alert  notification  window  (middle  green  window),  and 
the  sensor  display  (bottom  portion  of  display),  which  combines  information  from  active  radar, 
IFF,  NO  R,  and  RWR,  as  well  as  datalink  from  other  friendly  aircraft.  The  arrow  on  the  right 
side  of  the  display  indicates  the  pilot’s  presumed  current  attention  focus.  These  displays,  and 
their  associated  symbology,  are  described  in  detail  below. 
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HUD  Display  The  heads-up  display  combines  a  variety  of  navigation  and  sensor 
information  within  a  single  display.  The  top  scale  represents  the  heading  indicator  and  heading 
ruler.  The  vertical  scale  on  the  left  represents  the  airspeed  indicator,  showing  both  knots  and  a 
spinning  dial,  to  indicate  rate  of  change  in  airspeed.  Under  the  speed  indicator  is  the  MACH 
speed,  and  under  the  mach  speed  indicator  is  the  fuel  indicator,  showing  fuel  in  thousands  of 
pounds.  On  the  right  top  is  the  altimeter,  with  height  in  feet,  and  a  displacement,  surrounded  by 
another  spinning  dial,  again  to  indicate  rate  of  change  in  altitude.  At  the  bottom  is  a  clock, 
indicating  current  simulation  time.  A  red  box  near  center  indicates  the  location  of  the  current 
target. 

Communication  Window  Display  The  communication  window  displays  incoming 
communication  as  text  strings. 

Alert  Notification  Window  Display  The  alert  notification  window  displays  current 
alarms  and  notifications  as  text  strings. 
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Radar  Window  Display  The  radar  window  display  combines  information  from  the  on¬ 
board  sensors  (active  TWS  radar,  NCTR,  and  IFF).  The  own  aircraft  is  shown  in  the  center,  with 
the  concentric  rings  indicating  range  in  miles,  using  standard  radar  symbology.  The  contacts  are 
displayed  as  filled  circles  when  data  are  arriving  from  on-board  sensors,  and  as  hollow  circles 
when  data  are  arriving  via  datalink.  The  heading  of  the  individual  contacts  is  indicated  by  the 
associated  lines,  with  the  line  lengths  corresponding  to  the  contact  speed.  The  sensor  display  can 
also  show  radar  locks,  both  those  originating  from  the  friendly  aircraft,  and  those  originating 
from  hostile  or  unknown  radars,  and  appear  on  the  radar  as  inward-pointing  triangles  shown  on 
the  outer  circle  of  the  display.  Ground-based  radar  emitters  are  shown  as  rectangles  on  the 
outside  of  the  radar  display. 

Specific  contact  symbology  is  depicted  as  follows:  rectangular  ship  icon  means 
unknown/enemy;  circular  ship  icon  means  friendly;  green  ship  icon  means  friendly,  red  ship 
icon  mean  enemy,  yellow  ship  icon  means  unknown;  white  box  around  ship  icon  indicates  that 
user  is  targeting  that  ship.  Friendly  missiles  are  represented  by  white  dots,  with  a  radar-lock  line 
pointing  from  the  missile  icon  to  its  target. 

3.6.2  Analyst’s  GUI 

The  analyst’s  GUI  serves  three  functions: 

•  It  allows  specification  of  all  ABAIS  system  run-time  parameters,  including  task  script  editing 

and  selection,  adaptation  thresholds,  and  execution  monitoring  windows. 

•  It  allows  specification  of  all  necessary  background  pilot  information. 

•  It  allows  display  and  monitoring  of  ABAIS  simulation  and  run-time  data 

These  are  discussed  in  detail  below. 

The  ABAIS  system  run  begins  with  the  analyst  defining  the  necessary  system  parameters, 
specifying  any  relevant  pilot  information,  and  defining  the  task.  The  initial  ABAIS  menu  bar  is 
shown  in  figure  3. 6.2-1.  This  menu  provides  access  to  the  top  level  set  of  commands.  The  'File' 
and  'Window'  commands  support  the  definition  of  task  information,  task  script,  and  subject 
profile.  Simulation  Parameters  allow  the  specification  of  run-time  parameters,  controlling 
display  contents  and  configuration,  and  the  task  and  subject  profile  selection.  Commands 
associated  with  the  'Run'  menu  control  the  simulation  itself.  The  parameters  are  described  in 
more  detail  below. 
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Figure  3. 6. 2-1:  Initial  ABAIS  Menu  Bar 

ABAIS  System  Run-Time  Parameters  System  parameters  allow  the  analyst  to 
specify  the  ABAIS  GUI  configuration  during  a  system  run,  and  to  select  and  define  the  task 
script.  The  analyst  may  select  which  background  pilot  information  is  visible  during  execution 
(e.g.,  pilot  personality  profile,  training,  individual  history,  etc.).  The  analyst  also  selects  which 
rule  bases  s/he  wishes  to  see  during  execution,  and  specifies  the  adaptation  threshold,  which 
determines,  indirectly,  whether  or  not  adaptation  will  take  place.  Figures  3. 6.2-2  shows  screen 
shots  of  the  ABAIS  GUI  windows  that  allow  the  definition  of  these  run-time  parameters,  and 
indicates  specific  parameter  values  selected. 

Prior  to  a  simulation  run,  the  analyst  must  also  select  or  define  a  task  scenario  and  associated 
task  parameters.  This  is  done  by  defining  a  task  script,  which  specifies  which  events  occur 
during  the  task  and  their  timing.  Several  related  task  parameters  are  also  defined,  such  as  task 
type  and  complexity.  In  the  Phase  I  demonstration,  the  task  script  is  largely  fixed,  allowing  only 
a  limited  selection  of  specific  values  from  the  available  possibilities.  The  task  script  definition 
window  is  used  primarily  to  demonstrate  its  functionality  in  a  full-scope  ABAIS  (see  figure 
3.6.2-3).  In  a  full-scope  system,  each  of  the  task  script  cells  would  allow  the  selection  of 
specific  values  from  the  available  set,  thereby  allowing  the  definition  of  a  wide  variety  of 
experimental  scenarios. 

The  task  script  defines  the  entire  simulation  dynamic  environment.  That  is,  the  behavior  of 
both  the  lead  pilot  and  the  wingman,  the  lead  pilot's  aircraft  (i.e.,  any  instrument  failures,  etc.), 
enemy  aircraft,  and  any  weapon  deployment.  In  the  full-scope  implementation,  each  cell  in  the 
matrix  would  have  an  active  associated  listbox,  which  would  provide  the  set  of  possible  values 
for  that  cell. 

Background  Pilot  Information  Prior  to  a  run,  the  analyst  must  specify  background 
information  about  the  pilot.  This  is  a  necessary  component  of  the  type  of  individualized 
adaptation  implemented  by  the  ABAIS  system.  In  this  Phase  I  demonstration,  these  values  are 
entered  by  the  analyst.  In  a  full-scope  system,  some  of  these  parameters  might  be  entered  by  the 
pilot  (e.g.,  self-reports  and  individual  history  information),  gathered  during  training  tasks  (e.g., 
baseline  physiological  or  diagnostic  task  data),  or  collected  automatically  during  an  actual 
system  run  (e.g.,  actual  physiological  signals  or  diagnostic  task  results).  The  analyst  must 
specify  the  background  information  necessary  for  the  assessment  of  the  pilot's  affective  and 
belief  state,  and  must  define  the  pilot's  GUI  adaptation  profile.  This  profile  specifies  the  pilot's 
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Figure  3. 6. 2-2:  Analyst’s  GUI  Menus  Specifying  ABAIS 
System  Run-Time  Parameters 
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Figure  3. 6. 2-3:  Task  Script  Definition  Window 
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Figure  3. 6. 2-4:  Background  Pilot  Information: 
Information  Preference  Profile 


preferred  modality  and  other  UI  parameters  that  guide  the  selection  of  the  specific  adaptation 
method  (e.g.,  use  of  pilot's  preferred  sensory  modality  for  notifications,  preferred  icons  and 
display  formats,  etc.).  Figures  3. 6.2-4  and  3. 6.2-5  show  screen  shots  of  the  ABAIS  GUI 
windows  that  allow  specification  of  this  information. 
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Figure  3. 6. 2-5:  Background  Pilot  Information: 
Training,  Individual  History,  and  Personality 
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3.7  Simulation  Module 

To  demonstrate  the  ABAIS  adaptive  methodology  in  the  context  of  an  AF  sweep  task,  we 
integrated  the  core  ABAIS  architecture  with  a  flight  simulation  module.  Two  modes  of 
simulation  are  possible:  1)  pilot-as-user  mode,  where  the  user  actually  flies  the  aircraft  and 
interacts  with  a  simulated  environment  consisting  of  enemy  aircraft,  other  friendly  aircraft,  and 
missiles;  and  2)  analyst-as-user,  where  the  analyst  watches  a  scripted  task  and  monitors  the 
scripted  pilot's  performance,  as  well  as  the  rule-based  inferencing  results.  Due  to  the  precise 
timing  required  to  demonstrate  the  adaptation  GUI  changes,  the  emphasis  under  Phase  I  was  on 
developing  the  analyst-as-user  mode  and  the  associated  script-based  simulation,  which  allows 
precise  control  over  the  external  task  events  necessary  to  demonstrate  the  real-time  adaptation. 

The  simulation  module  output  is  shown  on  two  windows  representing  the  pilot’s  GUI,  as 
described  in  detail  in  section  3.7  above.  One  window  shows  the  simulated  cockpit  displays 
without  adaptation,  the  other  window  displays  the  ABAIS-derived  adaptations  on  the  same  set  of 
cockpit  displays.  During  a  simulation  run,  these  two  sets  of  cockpit  displays  are  synched  to  the 
same  moment  of  simulation-time,  allowing  the  analyst  to  compare  and  contrast  the  two  as  the 
adaptations  take  effect  during  the  run. 

A  third  window  contains  a  simple  set  of  run-time  controls  and  is  used  to  control  the 
simulation  during  the  run.  The  run-time  controls  allow  the  analyst  to  pause  the  simulation  at  any 
point  for  a  more  leisurely  study  of  the  simulation  displays,  and  of  the  rule-based  inferencing 
mediating  the  adaptation. 

Modeling  of  the  environment  is  done  within  the  simulation  proper,  with  the  dynamic 
characteristics  of  all  aircraft  (e.g.,  speed,  location,  heading),  and  the  instrumentation  of  the  pilot 
displays,  incremented  on  a  continuous  basis  as  time  in  the  simulation  progresses.  The  required 
aerodynamic  calculations,  simplified  for  scripted  control,  are  used  to  model  the  scenario  aircraft. 
Instrumentation  is  a  simulation-generated  reflection  of  the  environment,  with  communication, 
data  links  and  special  events  introduced  as  specified  by  the  task  script.  Scripted  events  take 
place  at  discrete  times,  modifying  the  behavior  of  the  friendly  and  enemy  pilots  or  introducing 
new  elements  into  the  simulation. 

Although  the  demonstration  analyst-as-user  mode  follows  a  precise  script,  the  simulation  is 
also  designed  to  allow  real-time  interactive  control  of  the  friendly  aircraft  by  the  user  and  non- 
scripted  response  by  the  enemy  aircraft. 
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4.  System  Operation  and  Demonstration 

The  objective  of  the  full-scope  ABAIS  system  is  to  provide  an  experimental  testbed 
implementing  the  generic  adaptive  methodology.  Such  a  testbed  would  facilitate  the  exploration 
of  a  variety  of  assessment  and  adaptation  strategies  across  a  range  of  task  scripts.  The  system 
would  support  both  the  definition  of  specific  task  scripts  and  rule-bases,  and  the  application  of 
these  rules  to  the  selected  task  script  to  demonstrate  the  system  adaptive  methodology. 

During  the  Phase  I  effort  a  limited  subset  of  this  functionality  was  developed,  focusing  on  the 
analyst's  graphical  user  interface,  to  provide  a  look-and-feel  of  the  envisioned  system.  In  this 
chapter  we  illustrate  the  ABAIS  system  by  demonstrating  its  functionality  in  the  context  of  the 
selected  scenarios.  The  ABAIS  demonstration  runs  include  the  analyst's  interaction  both  prior  to 
a  simulation  run,  where  the  analyst  selects  or  edits  a  task  script,  specifies  the  necessary  pilot 
information,  and  defines  the  system  run-time  parameters,  and  during  the  simulation  run,  where 
the  analyst  monitors  system  performance  via  a  customized  GUI  showing  the  selected  rule  bases 
and  pilot  profile  parameters.  ABAIS  system  functionality  is  demonstrated  by  illustrating  the 
inferencing  supporting  system  adaptation  in  the  context  of  the  demonstration  scenarios.  The 
scenarios  were  designed  to  demonstrate  multiple  points  at  which  a  particular  affective  state, 
personality  trait,  or  belief  can  bias  processing.  ABAIS  performance  is  illustrated  by  showing  the 
system  output,  in  terms  of  the  pilot  GUI’s,  for  several  pilot  profiles,  with  and  without  adaptation, 
as  determined  by  different  adaptation  thresholds.  The  pilot  profiles  vary  in  terms  of  personality 
characteristics,  individual  history,  training,  and  simulated  physiological  signals.  The  different 
pilot  profiles  thus  generate  different  affective  and  belief  states  at  different  points  of  the  scenario. 
The  pilot  profiles  also  specify  different  preferences  for  GUI  features,  resulting  in  different 
formats  of  the  information  provided  as  a  result  of  the  adaptation. 

The  demonstration  illustrates  the  ABAIS  system  functionality  across  all  stages  of  the 
adaptive  process.  The  specific  GUI  /  DSS  adaptations  are  derived  from  a  sequence  of  instantiated 
rules  from  the  four  ABAIS  modules.  The  output  of  the  GUI  /  DSS  adaptation  recommendations 
is  shown  on  the  corresponding  cockpit  instruments. 

The  ABAIS  implementation  supports  two  modes  of  system  operation:  1)  pilot-as-user 
mode,  where  the  user  actually  flies  the  aircraft  and  interacts  with  a  simulated  environment 
consisting  of  enemy  aircraft,  other  friendly  aircraft,  and  missiles;  and  2)  analyst-as-user,  where 
the  analyst  watches  a  scripted  task  and  monitors  the  scripted  pilot's  performance  as  well  as  the 
rule-based  inferencing  results.  These  modes  are  not  entirely  mutually  exclusive.  In  other  words, 
the  analyst  and  /  or  the  pilot  can  specify  pilot  and  system  parameters,  the  pilot  can  they  "fly''  the 
aircraft,  and  both  can  watch  the  system  output  on  simulated  cockpit  instruments  and  ABAIS 
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system  monitoring  windows.  Due  to  the  precise  coordination  required  to  fly  the  demonstration 
script,  the  ABAIS  performance  is  demonstrated  in  the  analyst-as-user  mode,  where  the  aircraft  is 
controlled  by  a  script  that  emulates  the  desired  pilot  behavior. 

This  chapter  is  organized  as  follows.  Section  4.1  discusses  the  affective  state,  personality 
characteristics,  and  beliefs  chosen  for  demonstration  of  the  adaptive  methodology.  Section  4.2 
describes  the  demonstration  scenarios.  Section  4.3  illustrates  the  sequence  of  steps  and  the 
ABAIS  adaptation,  via  a  series  of  screenshots  from  the  ABAIS  GUI. 

4.1  Selection  of  Affective  States,  Personality  Characteristics,  and  Belief 
States 

Anxiety  was  selected  as  the  primary  affect  and  obsessiveness  as  the  primary  personality  trait 
for  the  Phase  I  effort.  This  selection  was  made  in  part  because  much  is  known  about  both  of 
these  factors  and  their  effect  on  performance,  and  in  part  because  both  factors  can  play  a  role  in 
the  types  of  dynamic,  real-time  environments  that  are  of  interest  to  the  Air  Force.  Although  the 
fighter  pilot  population  is  certainly  not  in  general  characterized  by  a  propensity  towards  anxiety, 
anxiety  can  and  does  occur  during  combat  and  can  severely  influence  performance,  as  evidenced 
by  a  number  of  documented  aviation  mishaps. 

Obsessiveness  represents  a  combination  of  anxiety  and  one  of  the  “Big  5’  personality  traits: 
the  conscientiousness  dimension.  The  effects  of  obsessiveness  on  performance  have  also  been 
studied  and  provide  interesting  possibilities  for  adaptation. 

The  pilot’s  relevant  belief  set  is  defined  by  the  possible  set  of  situations  as  described  in  table 
3.2.2. 1-1  in  section  3.2.2  above.  In  the  specific  context  of  the  demonstration  scenarios,  the 
possible  belief  states  include  identity  of  contacts  (unknown,  hostile,  friendly),  status  of  own 
aircraft  (under  attack  or  not),  status  of  weapon  deployment  (attacking  or  not).  This  set  of  beliefs 
was  selected  because  it  adequately  represents  the  possible  situations  and  likely  actions  in  the 
constrained  behavior  space  of  the  demonstration  scenario. 

4.2  Demonstration  Scenarios 

Key  element  of  a  successful  ABAIS  system  demonstration  was  the  selection  of  an 
appropriate  demonstration  scenario.  The  scenario  should  satisfy  the  following  criteria: 

•  Adequacy  The  scenario  should  provide  adequate  opportunities  for  demonstrating  key 

ABAIS  affect,  personality,  and  belief  assessment  and  adaptation  features. 

•  Credibility  The  scenario  should  provide  a  credible  context  within  which  to  effectively 

demonstrate  the  ABAIS  adaptation  features.  The  scenario  situations  should  be  capable  of 
inducing  varying  levels  of  anxiety  and  provide  opportunity  for  performance  biases  due  to 
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selected  personality  traits  and  likely  beliefs.  The  scenario  should  also  provide  credible 
opportunities  for  prediction  of  likely  performance  biases  and  for  subsequent  adaptation. 

•  Feasibility  The  scenario  implementation  must  be  feasible  within  the  time  available  for  the 

Phase  I  effort. 

•  Relevance  Last,  but  not  least,  the  selected  scenario  should  demonstrate  ABAIS  in  an 

environment  of  interest  to  the  USAF,  and  demonstrate  its  ability  to  improve  decision 
making  performance. 

In  addition,  the  task  environment  as  a  whole  must  provide  relevant  data  in  an  electronic 
format,  to  facilitate  the  type  of  inferencing  necessary  for  implementing  the  user  assessment  and 
adaptation. 

To  satisfy  these  requirements,  we  have  selected  a  fighter  pilot  context,  and  a  defined  a  sweep 
task  scenario  within  this  context.  A  sweep  task  represents  a  situation  where  stress  levels  are 
likely  to  be  high,  accurate  situation  assessment  is  critical,  and  consequences  of  affect-  and 
belief-induced  performance  biases  can  be  catastrophic,  or  at  best  expensive.  Two  sequential 
scenario  segments  within  this  general  contexts  were  defined,  each  designed  to  demonstrate 
adaptation  to  a  particular  combination  of  affect,  personality,  or  belief-induced  performance  bias. 

The  scenario  sequence  involves  two  friendly  fighter  aircraft  engaging  several  unknown  and 
presumed  hostile  aircraft  in  the  course  of  a  sweep  task.  Multiple  variations  within  this  basic 
structure  provide  opportunities  to  demonstrate  varying  affective  states  (e.g.,  low  vs.  high 
anxiety),  varying  expressions  of  particular  personality  traits  (e.g.,  low  vs.  high  obsessive  / 
conscientious),  and  differences  in  individual  history  and  training  with  consequent  variations  in 
performance.  The  aircraft  instrumentation  is  loosely  modeled  on  the  F-15  fighter  aircraft  which 
is  assumed  to  be  equipped  with  a  HUD  and  datalink  connections  between  the  lead,  wingman,  and 
an  AWACS.  Critical  data  are  thus  shared  between  the  lead,  wingman,  and  the  AWACS  pilots. 

The  ABAIS  system  is  demonstrated  in  the  context  of  the  lead's  cockpit  and  is  assumed  to  be 
tightly  coupled  with  the  detection  instruments  and  the  radar  and  HUD  displays.  The  ABAIS 
system  thus  has  access  to  any  data  appearing  on  the  radar,  whether  obtained  directly  from  the 
aircraft  sensors,  or  indirectly  from  wingman  or  AWACS  via  datalink.  ABAIS  uses  these  data  as 
the  basis  for  its  dynamic  inferencing  when  assessing  the  pilot's  affective  and  belief  state, 
inferring  the  adaptation  strategy,  and  implementing  the  GUI  adaptation. 

The  two  scenario  segments  are  described  in  detail  below.  Each  description  is  followed  by  a 
highlighting  of  the  features  that  are  most  relevant  for  the  ABAIS  adaptation  demonstration. 
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4.2.1  Scenario  #1  -  Anxiety-Induced  Attention  and  Perception  Bias 

The  scenario  description  below  is  excerpted  from  Shaw  (1998a). 

Two  aircraft  are  conducting  a  sweep  mission  under  the  control  of  AWACS.  The  aircraft  are 
equipped  with  track-while-scan  (TWS)  air-to-air  radar,  radar  warning  receiver  (RWR),  IFF 
interrogation  capability  for  friendly  aircraft,  a  non-cooperative  target  recognition  (NCTR) 
capability  providing  aircraft  type  identification,  and  air-to-air  datalink  among  both  fighters  and 
AWACS. 

This  is  the  first  day  of  the  air  campaign  and  the  first  combat  mission  for  both  pilots,  so  the 
anxiety  level  is  high.  The  pilots  have  been  briefed  to  expect  strong  enemy  fighter  opposition. 
Enemy  fighters  have  long-range  radar- guided  missiles  of  capability  similar  to  that  of  the  F-15s. 
There  are  also  likely  to  be  aircraft  of  the  allied  coalition  in  the  area,  so  either  an  electronic-  or 
visual  identification  or  clearance  from  AWACS  will  be  required  before  any  attacks. 

AWACS  notifies  lead  pilot  by  voice  radio  of  unknown  number  of  “probable  hostile  contacts” 
lOOnm  north,  headed  south  at  high  speed,  medium  altitude.  A  datalink  target  generated  by 
AWACS  is  displayed  on  both  fighter  radars  beyond  normal  radar  contact  range.  The  fighters 
vector  north  toward  the  contacts  and  attempt  to  acquire  radar  contacts. 

At  70nm  range  AWACS  notifies  that  contacts  are  “presumed  hostile  fighters”,  continuing  on 
same  course  at  same  heading,  altitude,  airspeed  and  that  the  friendly  aircraft  are  cleared  to  fire 
with  a  positive  electronic  identification  (EID)  (i.e.,  IFF  or  NCTR) .  Same  information  shows  up 
on  datalink  as  before. 

At  50nm  the  lead  pilot  obtains  radar  contact  with  a  single  target  on  the  nose.  This 
information  is  provided  to  the  AWACS  and  the  wingman,  both  verbally  and  automatically  via 
datalink.  The  wingman  does  not  yet  have  radar  contact  but  gets  a  radar  display  from  information 
provided  by  lead  via  datalink,  indicated  as  datalink  targets  by  hollow  icons,  in  contrast  to  actual 
sensor  contacts  which  are  solid. 

Lead  pilot  initiates  IFF  interrogation  of  target  and  gets  no  reply.  This  may  indicate  either  a 
faulty  IFF  interrogator,  inoperative  friendly  target  IFF,  or  hostile  aircraft.  At  40nm  the  lead's 
radar  identifies  multiple  aircraft  in  the  target  group:  first  2,  then  4  separate  aircraft. 
Simultaneously,  the  wingman  calls  radar  contact  with  the  group,  also  indicating  that  he  sees  4 
aircraft.  Another  IFF  interrogation  is  attempted  with  the  same  negative  results.  The  lead  instructs 
his  wingman  to  attempt  IFF  interrogation. 
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At  35nm,  approaching  maximum  missile4  firing  range  for  both  the  F-15s  and  the  presumed 
enemy  fighters,  the  wingman  reports  “sorted”  on  the  target  group,  indicating  that  he  has  selected 
the  targets  in  the  group  that  he  has  been  instructed  in  the  brief  to  target,  and  negative  IFF 
interrogation. 

The  automated  NCTRs  of  both  aircraft  have  been  operating,  but  have  not  yet  resolved  the 
target  aircraft  type.  Up  to  now  the  targets  are  identified  as  “Unknown  Type.” 

At  30nm,  just  as  maximum  firing  range  is  reached,  the  lead's  RWR  indicates  an  “Unknown 
Air-to-Air  Radar”  lock  on  the  nose,  followed  shortly  by  indication  of  an  enemy  SAM  radar  lock- 
on  on  the  left-front  quarter.  This  information  is  not  transmitted  automatically  via  data  link  to 
either  the  AWACS  or  the  wingman. 

The  wingman  calls,  in  a  high-pitched  voice,  that  he  too  is  targeted  by  the  SAM.  Through  the 
blaring  RWR  audio  the  lead  hears  the  AWACS  controller  issue  a  “cleared  to  fire.”  The  last  part 
of  the  transmission  is  difficult  to  understand  because  the  wingman  makes  a  simultaneous 
transmission  on  the  other  radio  notifying  the  lead  that  his  NCTR  has  just  provided  a  “Friendly” 
ID  on  one  of  the  targets  in  the  group.  Which  exact  target  is  not  specified  and  the  implication  is 
that  all  other  targets  in  the  group  are  also  friendly. 

The  above  information  is  also  provided  to  lead’s  radar  automatically  via  datalink,  in  addition 
to  the  voice  radio  communication  from  wingman.  The  lead  is  busy  “centering  the  dot”  for  his 
missile  shot,  attempting  to  target  the  enemy  leader  or  the  first  enemy  on  his  side,  depending  on 
the  established  sort,  and  does  not  grasp  the  significance  of  the  wingman’ s  radio  call.  As  the  lead 
aircraft's  first  missile  leaves  the  rail,  the  wingman  calls  “Cease  fire,  cease  fire!” 

4. 2. 1.1  Key  Features  of  Scenario  # 1 

This  scenario  was  designed  to  demonstrate  adaptation  to  the  pilot's  heightened  state  of 
anxiety  and  thus  provides  multiple  opportunities  for  demonstrating  the  effect  of  anxiety  on 
attention  and  perception.  These  include  the  presence  threatening  stimuli  (i.e.,  presumed  hostile 
aircraft,  RWR  warnings),  and  the  presence  of  multiple  unknown  stimuli  (e.g.,  unknown  aircraft 
approaching,  RWR  warnings).  The  large  number  of  unknown  (i.e.,  ambiguous)  signals  provide 
both  a  source  of  anxiety,  potential  for  anxiety-  and  belief-induced  performance  bias,  and 
consequent  opportunities  for  adaptation.  The  ambiguous  signals  include: 


'’The  friendly  missiles  are  assumed  to  be  AMRAAM  missiles,  which  are  guided  by  the  fighter's  radar  for  the 
first  part  of  a  long-range  shot.  Once  the  missile  is  close  to  its  target,  it  becomes  "active"  (turns  on  its  own  radar)  and 
tracks  the  target  independently.  Enemy  missiles  are  assumed  to  be  the  AA-12  or  equivalent,  which  are  similar  in 
capability  and  operation  to  the  AMRAAM. 
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•  Both  friendly  aircraft  are  being  targeted  by  a  SAM 

•  A  number  of  unknown  but  presumed  hostile  aircraft  are  approaching 

•  The  lead  aircraft  indicates  an  unknown  air-to-air  radar  lock 

•  Unknown  but  presumed  hostile  aircraft 

•  Unknown  NCTR 

•  Unknown  IFF  signals  (no  response) 

•  Unknown  RWR 

The  most  critical  moment  in  this  scenario  occurs  when  the  lead  is  about  to  launch  a  weapon 
at  an  approaching  friendly  aircraft  which  he  has  mistaken  as  hostile.  The  last  sequence  of  events 
is  as  follows: 

1.  Lead  pilot  hears  the  AWACS  radio  "Cleared  to  fire"  transmission 

2.  Simultaneously  the  wingman  radios  that  his  NCTR  has  just  provided  a  friendly  ID  on  one 
of  the  targets  in  the  approaching,  'presumed  hostile'  aircraft  -  this  critical  transmission  is  not 
heard  by  the  lead. 

3.  The  wingman ’s'  friendly  ID  NCTR  info  is  provided  to  the  lead  via  a  datalink  and  appears 
on  his  radar. 

4.  Not  expecting  this  information,  and  focusing  on  centering  the  dot,  the  lead  fails  to  take 
notice  of  this  and  fires  a  missile  at  the  friendly  aircraft  due  to  his  heightened  anxiety  level. 

The  ABAIS  objective  is  to  prevent  the  fratricide,  by  detecting  its  possibility  and  adapting  the 
pilot’s  cockpit  GUI  to  assure  that  the  pilot  notices  the  changed  status  of  the  unknown  contact 
from  presumed  hostile  to  friendly. 

4.2.2  Scenario  #2  -  Obsessiveness-Induced  Performance  Bias 

The  scenario  description  below  is  excerpted  from  Shaw  (1998b). 

Two  aircraft  are  conducting  a  sweep  mission  under  the  control  of  AWACS.  Aircraft  are 
equipped  with  track-while-scan  (TWS)  air-to-air  radar,  radar  warning  receiver  (RWR),  IFF 
interrogation  capability  for  friendly  aircraft,  a  non-cooperative  target  recognition  (NCTR) 
capability  providing  aircraft  type  identification,  and  air-to-air  datalink  among  both  fighters  and 
AWACS. 

This  is  the  second  week  of  the  air  campaign  and  the  second  combat  mission  for  the  Lead 
pilot,  while  the  Wingman  has  completed  several  combat  missions  and  been  credited  with  one 
confirmed  “Kill.”  On  the  Lead’s  first  combat  mission,  he  had  fired  a  missile  at  a  friendly  aircraft 
by  mistake.  Ambiguous  and  inconclusive  electronic  identification  (EID)  data  had  contributed  to 
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this  mishap.  Following  that  mission  the  pilot  had  been  grounded  and  forced  to  endure  a  formal 
Board  of  Inquiry.  He  was  absolved  of  total  responsibility  for  the  incident,  primarily  because  he 
had  been  given  firing  clearance  from  his  AWACS  controller,  but  the  pilot  still  had  received  a 
“letter  of  reprimand”  for  his  premature  decision  to  fire  without  a  confirmed  Hostile  EID. 

The  pilots  have  been  briefed  that  the  enemy  fighter  force  had  been  severely  weakened 
through  attrition  and  evacuation  to  a  neighboring  country,  so  strong  opposition  was  not  expected. 
Some  remaining  enemy  fighters,  however,  may  still  have  long-range  radar-guided  missiles  of 
capability  similar  to  that  of  the  F-15s.  There  are  also  likely  to  be  aircraft  of  the  Allied  Coalition 
in  the  area,  so  either  an  electronic-  or  visual  identification  will  again  be  required  before  any 
attacks. 

AWACS  notifies  lead  pilot  by  voice  radio  of  “possible  hostile  contacts”  lOOnm  northwest, 
headed  southwest  at  high  speed,  low  altitude.  The  Lead  confirms,  “Roger,  Limbo  01  Flight 
understands  possible  hostile  contacts  330,  lOOnm,  low  altitude,  high  aspect.  Turning  330, 
searching.  A  datalink  target  generated  by  AWACS  is  displayed  on  both  fighter  radars  beyond 
normal  radar  contact  range.  The  fighters  vector  330  degrees  toward  the  contacts  and  attempt  to 
acquire  them  on  radar. 

At  70nm  range  AWACS  notifies  that  contacts  “presumed  hostile  fighters”  and  that  the 
friendly  aircraft  are  cleared  to  fire  with  a  positive  electronic  identification  (EID).  The  Lead 
responds.  Limbo  01  Flight  understands  targets  presumed  hostile  and  cleared  to  fire  with  positive 
EID.” 

At  50nm  the  lead  obtains  radar  contact  with  a  single  target  on  the  nose.  This  information  is 
provided  to  the  AWACS  and  the  wing  both  verbally  and  automatically  via  datalink.  The  Wing 
does  not  yet  have  radar  contact. 

Lead  initiates  IFF  interrogation  of  the  target  and  gets  no  reply.  This  may  indicate  either 
faulty  IFF  interrogator,  inoperative  friendly  target  IFF,  or  hostile  aircraft.  IFF  interrogation  is 
again  attempted  at  45nm  range. 

At  40nm  the  lead's  radar  identifies  two  aircraft  in  the  target  group,  echeloned  west. 
Simultaneously,  the  Wingman  calls  radar  contact  with  the  group.  Another  IFF  interrogation  is 
attempted  with  the  same  negative  results.  The  Lead  acknowledges  the  Wingman’ s  radar  contact 
and  instructs  him  to  attempt  IFF  interrogation. 

At  35nm,  approaching  maximum  missile  firing  range  for  both  friendly  aircraft  and  the 
presumed  enemy  fighters,  the  Wingman  reports  “sorted”  on  the  target  and  negative  IFF 
interrogation.  The  Lead  calls  the  Wingman  to  “Confirm  azimuth  sort;”  and  the  Wingman  replies 
in  the  affirmative.  The  Leader  attempts  another  IFF  interrogation  and  rechecks  his  weapons 
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switches,  which  were  set  some  time  previously  when  entering  hostile  airspace.  The  automated 
NCTRs  of  both  friendly  aircraft  have  been  operating,  but  have  not  yet  resolved  the  target  aircraft 
type.  Up  to  now  the  targets  are  identified  as  “Unknown  Type.” 

At  30nm,  just  as  maximum  firing  range  is  reached,  the  lead’s  RWR  indicates  an  “Unknown 
Air-to-Air  Radar”  lock  on  the  nose.  The  Wingman  calls  on  VHF  tactical  frequency  that  he  too  is 
“spiked.”  Near  simultaneously,  on  the  UHF  receiver,  Lead  hears  the  AWACS  controller  issue  a 
“cleared  to  fire  with  positive  ID.”  The  last  part  of  the  transmission  is  difficult  to  understand 
because  it  is  “stepped  on”  by  the  Wingman’s  radio  call,  so  the  Lead  responds  on  VHF,  “Roger  2, 
understand  spiked.”  He  then  queries  the  AWACS  controller  on  UHF,  “Limbo  01  Flight, 
understand  cleared  to  fire  with  positive  EID.  Negative  EID  at  this  time.”  Simultaneously,  the 
Lead  attempts  another  IFF  interrogation. 

At  25nm  the  Wingman  reports  Hostile  EID  on  one  target  aircraft.  The  Leader  responds, 
“Roger  2,  understand  Hostile.  Negative  ID  here.”  The  Lead  attempts  another  IFF  interrogation. 
Although  technically  now  cleared  to  fire,  the  Leader  hesitates  in  order  to  confirm  the  ID  with  his 
own  EID  systems.  IFF  interrogation  is  repeated  again  at  20nm,  just  as  the  Wingman  repeats, 
with  added  urgency,  “Limbo  01,  repeat!  Limbo  02  has  POSITIVE  Hostile  ID  on  target. 
Recommend  FOX  3!”  Before  the  Leader  can  respond,  he  receives  a  blaring  audio  warning  from 
his  RWR  indicating  a  hostile  air-to-air  missile  targeted  at  this  aircraft. 

4.2. 2. 2  Key  Features  of  Scenario  #2 

Scenario  #2  was  designed  to  provide  opportunities  for  adaptation  to  the  pilot's  heightened 
obsessiveness,  and  thus  provides  multiple  opportunities  for  demonstrating  the  effect  of 
obsessiveness  on  decision  making.  The  primary  effect  here  is  excessive  checking  behavior  and 
delayed  decision  making,  which  manifest  itself  in  repeated  and  lengthy  confirmations  of 
communication  transmissions.  Again,  as  with  scenario  #1  above,  the  large  number  of  unknown 
or  ambiguous  stimuli  provide  ample  opportunities  for  expressions  of  the  checking  behaviors. 

•  Possible  hostile  contacts  at  100  nm  result  in  lengthy  echoing  of  information  by  lead. 

•  Presumed  hostile  aircraft  at  70  nm  and  clearance  to  fire  results  in  another  lengthy  echoing  of 

status  by  lead. 

•  At  35  nm,  when  wingman  reports  “sorted”,  lead  asks  for  unnecessary  confirmation  of  sort 

type,  and  repeats  IFF  interrogation.  The  lead  also  rechecks  his  weapons. 

•  At  30  nm,  which  is  also  the  maximum  firing  range,  both  lead  and  wingman  are  spiked  by  a 

presumed  enemy  radar  and  lead  engages  in  yet  another  lengthy  confirmation  of  the  ‘cleared 

to  fire’  order  from  the  AWACS. 

•  Although  cleared  to  fire,  at  25  nm  lead  attempts  yet  another  IFF.  Simultaneously  wingman 
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reports  positive  hostile  id  on  one  target  aircraft.  Leader  responds,  indicating  that  his  own  id 
remains  negative. 

•  At  20,  wingman  repeats  his  positive  id  and  suggests  that  lead  deploy  weapon. 
Simultaneously,  the  lead  receives  an  RWR  warning  that  he  is  being  attacked  by  an  air-to- 
air  missile. 

There  are  thus  a  number  of  points  during  this  scenario  where  extensive  checking  behavior 
occurs,  delaying  the  pilot’s  decision  to  deploy  the  aircraft  weapon,  and  culminating  in  the  most 
critical  moment  in  this  scenario  which  occurs  when  the  lead  aircraft  is  targeted  by  an  enemy 
aircraft. 

The  ABAIS  objective  is  to  prevent  this  situation,  by  detecting  its  possibility  and  adapting  the 
pilot’s  cockpit  GUI  to  attempt  the  reduce  the  checking  behavior  and  induce  the  pilot  to  initiate 
timely  weapons  deployment. 

4.3  System  Demonstration:  Scenario  #1 

This  section  describes  in  detail  the  ABAIS  simulation  for  scenario  #1.  The  ABAIS  system 
performance  is  demonstrated  by  describing  the  rule-based  reasoning  comprising  the  adaptation 
sequence,  starting  with  user  assessment  and  culminating  with  the  GUI  /  DSS  adaptations.  The 
scenario  is  divided  into  discrete  time  frames,  and  ABAIS  inferencing  and  resulting  GUI  output 
on  the  cockpit  displays  are  shown  for  each  time  frame.  The  ABAIS  simulation  walk-throu°h 
also  illustrates  the  flexibility  of  the  adaptation,  which  is  controlled  by:  1)  specific  settings  of  the 
pilot  background  information  profile  to  generate  varying  levels  of  anxiety  (see  tables  4.3-1  and 
43-2);  2)  thresholds  controlling  the  level  of  anxiety  required  before  adaptation  is  triggered;  and 
3)  specific  pilot  preference  profiles,  representing  different  the  pilot  information  presentation 
preferences  (see  table  43-3).  These  parameters  give  rise  to  a  series  of  variations,  which  differ  in 
the  number  and  types  of  adaptations  suggested,  as  well  as  the  specific  formats  of  these 
adaptations  on  the  pilot  GUI. 

The  simulation  run  is  presented  as  a  series  of  discrete  frames,  where  each  frame  corresponds 
to  a  specific  time  segment  of  the  scenario  (see  tables  4.3-4  and  4.3-5).  Both  affect  and  belief 
states  are  derived  at  each  point  of  the  scenario,  from  both  static  factors  (c.g.,  personality 
structure,  training,  individual  history)  and  dynamic  factors  (e.g.,  current  number  and  type  of 
hostile  or  unknown  aircraft,  etc.).  The  rules  used  to  perform  the  user  assessment  are  shown  for 
each  frame. 

Two  pilot  profiles  are  specified  for  the  demonstration:  one  representing  an  anxiety-prone 
(high-anxious)  pilot,  the  other  representing  an  anxiety  tolerant  (low-anxious)  pilot.  These 
profiles  are  shown  in  tables  4.3-1  and  4.3-2.  Briefly,  the  low-anxious  pilot  is  more  highly  trained, 
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has  a  more  stable  personality  structure,  has  less  reactive  heart  rate  measures,  and  an  individual 
history  that  does  not  predispose  him  to  anxiety.  In  contrast  to  this,  the  high-anxious  pilot  has 
less  training  and  recency,  has  a  less  stable  personality  structure  as  reflected  in  both  a  lower  score 
on  the  emotional  stability  scale  and  a  more  reactive  physiological  profile,  and  has  an  individual 
history  that  predisposes  him  to  anxiety  in  particular  situations. 

In  addition  to  specifying  background  information  allowing  the  assessment  of  the  pilot's 
affective  and  belief  states,  pilot  profiles  also  allow  the  specification  of  GUI  presentation 
preferences.  Thus  the  same  recommended  adaptation  (e.g.,  "Enhance  unknown  target  display") 
can  appear  in  multiple  formats,  depending  on  the  pilot's  presentation  preference  (e.g.,  highlight 
icon  in  a  particular  color,  change  icon  size,  generate  an  auditory  notification,  etc.).  Examples  of 
presentation  profiles  used  for  the  demonstration  are  shown  in  table  4.3-3. 

Table  4.3-1:  Anxiety  Tolerant  (Low-Anxious)  Pilot  Profile 


Personality  factors 

Emotional  stability  =  high 

Training  /  Proficiency  factors 

air  combat  hours  =  high 

air  combat  exercises  hours  =high 

training  hours  =  high 

recency  of  air  combat  hours  =  high 

recency  of  air  combat  exercises  hours  =  high 

training  hours  =  high 

Specific  events  in  individual  history  contributing  to  anxiety 

SAM  lock 
Centering  the  dot 


Table  4.3-2:  Anxiety  Prone  (High-Anxious)  Pilot  Profile 


Personality  factors 

Emotional  stability  =  low 

Training  /  Proficiency  factors 

air  combat  hours  =  low 

air  combat  exercises  hours  =  low 

training  hours  =  low 

recency  of  air  combat  hours  =  low 

recency  of  air  combat  exercises  hours  =  low 

training  hours  =  low 

Specific  events  in  individual  history  contributing  to  anxiety 

Cleared  to  fire 
Unknown  type  from  NCTR 
Unsuccessful  IFF  interrogation 
Unknown  RWR  radar  lock 
SAM  lock 

"Centering  the  dot’1 _ _ 
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Table  4.3-3:  Examples  of  Different  Pilot  Presentation  Preference  Profiles 


Profile  #1 

Enhance  visibility  by  changing:  color 

Preferred  color  for  alarms:  red  solid 

Alarm  notification:  visual 

Attention  capture  preference:  display  over  foveal  area 


Profile  #2 

Enhance  visibility  by  changing:  size 

Preferred  color  for  alarms :  red  outline 

Alarm  notification:  visual 

Attention  capture  preference:  display  arrow  pointing  to  icon 

Profile  #3 

Enhance  visibility  by  changing:  color  and  size 
Preferred  color  for  alarms:  red  solid 

Alarm  notification:  visual  and  auditory 

Attentign_capture^reference:  movement  at  periphery _ 

The  scenario  is  summarized  in  two  tables,  highlighting  different  aspects  of  the 
demonstration.  These  tables  provide  overview  of  ABAIS  processing  during  the  demonstration 
scenario,  indicating  critical  external  events  and  corresponding  ABAIS  processing  for  each  time 
frame  of  the  script.  Table  4.3-4  shows  the  incoming  data  which  serve  a  primary  cues  for  belief 
derivation  and  also  as  input  to  the  affect  assessment  rule  base.  This  table  also  indicates  the  pilot's 
physiological  signals  (i.e.,  heart  rate  measures)  and  shows  the  ABAIS  inferred  belief  and 
affective  state  for  both  the  low  and  the  high-anxious  pilot.  Table  4.3-5  summarizes  the  scenario 
again,  this  time  showing  the  system  output  for  the  high-anxious  pilot  and  indicating  the  inferred 
affect  and  belief  state,  predicted  impact  of  these  states  on  performance,  selected  adaptation 
strategy,  and  specific  GUI  /  DSS  modifications  implementing  this  strategy,  and  specific  format 
of  the  GUI  modifications  as  defined  by  the  pilot  presentation  preference  profile.  The  actual 
pilot  s  GUI  (selected  cockpit  instruments),  and  the  analyst  GUI  for  the  last  critical  frames  where 
adaptations  occur,  are  shown  in  ABAIS  screenshots.  Each  of  these  frames  thus  shows  the  analyst 
GUI  with  the  selected  information  (e.g.,  task  script  events,  rule  base  monitors,  pilot  physiological 
signals,  etc.),  cockpit  instruments  indicating  the  incoming  data  on  the  corresponding  displays 
(e.g.,  approaching  hostile  targets,  etc.),  the  inferred  pilot  affective  and  belief  states,  and  the  GUI 
adaptations,  if  any. 
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Table  4.3-4:  Summary  of  Demonstration  Scenario:  Incoming  Cues  and 
Pilot's  Physiological  Signals,  Inferred  Belief  and  Affective  States  for  Low 

and  High-Anxious  Pilot 


Time 

Frame 

New  data 
/  cues 

Heart 

rate 

signals 

Inferre 
d  Belief 

Affectiv 
e  State 

Low 

anxious 

High 

anxious 

Low 

anxious 

High 

anxious 

Low 

anxious 

High 

anxious 

1: 100 
rnn 

AWACS: 

probable 

hostile 

contacts 

HR  nml 

HR  nml 

Unknown 

contacts 

Unknown 

contacts 

low 

low 

2:  70 
mn 

AWACS: 

probable 

hostile 

contacts; 

cleared  to 
fire  w  /  pos 
HD 

HR  nml 

HR  inc 

Unknown 

contacts 

Hostile 

contacts 

low 

low-med 

3:  50 
nm 

own:  active 
radar  single 
contact; 

NCTRunk; 
attempt  IFF 

HR  nml 

HR  inc 

Probable 

hostile 

contacts 

Hostile 

contacts 

low 

low-med 

4:  40 
nm 

own:  active 
radar  4 
targets; 
NCTRunk 

HR  inc 

HR  inc 

Probable 

hostile 

contacts 

Hostile 

contacts 

low 

med 

5:  37 
nm 

NCTRunk; 

attempt  IFF 
-  no  reply; 

HR  nml 

HR  inc 

Probable 

hostile 

contacts 

Hostile 

contacts 

low 

med 

6:  35 
nm 

wing: 

negative 

IFF 

own: 

NCTRunk 

HR  nml 

HR  inc 

Probable 

hostile 

contacts 

Hostile 

contacts 

low 

med 

7:  30 
nm 

own: 

RWRunk 
air-to-air 
radar  lock; 
hostile 

SAM  lock 

HR  inc 

HR  inc 

Hostile 

contacts 

Hostile 

contacts; 

Under 

attack 

low-med 

med-high 
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Table  4.3-4:  Summary  of  Demonstration  Scenario:  Incoming  Cues  and 
Pilot's  Physiological  Signals,  Inferred  Belief  and  Affective  States  for  Low 

and  High-Anxious  Pilot  (cont.) 


Time 

Frame 

New  data 
/  cues 

Heart 

rate 

signals 

Inferred 

Belief 

Affectiv 
e  State 

Low 

anxious 

High 

anxious 

Low 

anxious 

High 

anxious 

Low 

anxious 

High 

anxious 

8:  < 

30nm 

own: 

RWRunk 
air-to-air 
radar  lock 

AWACS: 
cleared  to 
fire 

wing:  unk 
air-to-air 
radar  lock 

HR  inc 

HR  inc 

Hostile 

contacts 

Hostile 

contacts; 

Under 

attack 

low-med 

med-high 

9: 

«30nm 

own: 

RWRunk 
air-to-air 
radar  lock; 

centerin 
g  the  dot 

AWACS: 
cleared  to 
fire 

wing: 

friendly  ID 
on  target 

HR  inc 

HR  max 

Hostile 

contacts; 

Under 

attack 

Hostile 

contacts; 

Under 

attack 

low-med 

high 

10:  «< 
30 

own: 

low  anx  - 
notice 
friendly 
ID 

high 
anx -  fire 
missile 

wing: 

"Cease  fire, 
Cease  fire" 

HR  inc 

HR  max 

Friendly 

contacts 

Hostile 

contacts; 

Under 

attack; 

Attacking 

low-med 

high 
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Table  4.3-5:  Summary  of  Demonstration  Scenario: 

Inferred  Belief  and  Affective  States  and  Predicted  Influence,  Compensatory 

Strategy,  and  GUI  /  DSS  Adaptation 


Time 

Frame 

Inferred 

Belief 

Affectiv 
e  State 

Predicted 

Impact 

Selected 

Strategy 

GUI /  DSS 
Adaptation 

Anxiety 

1:  100  nm 

Unknown 

contacts 

low 

2:  70  nm 

Hostile 

contacts 

low-med 

3:  50  nm 

Hostile 

contacts 

low-med 

4:  40  nm 

Hostile 

contacts 

med 

5:  37  nm 

Hostile 

contacts 

med 

5:  35  nm 

Hostile 

contacts 

med 

6:  30  nm 

Hostile 

contacts 

med-high 

Threat  estimation 
bias  (unknown 
NCTR;  neg  IFF) 
selects  threat 
interpretation  (i.e., 
hostile  contacts) 

Emphasize 
unknown 
target  status 

Subject  Profile  #1: 
Enlarge  icon  size 
Subject  Profile  #2: 

Highlight  icon  w/ 
color 

7:  <30nm 

Hostile 

contacts; 

Under  attack 

med-high 

Threat  estimation 
bias  (unknown 
RWR)  selects 
threat 

interpretation  (i.e., 
under  attack) 

Emphasize 
unknown 
target  status; 

Subject  Profile  #1: 
Enlarge  icon  size 

Subject  Profile  #2: 

Highlight  icon  w/ 
color 

Both  subjects: 
Display  that 
aircraft  is  NOT 
within  range  of 
enemy  missiles 
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Table  4.3-5:  Summary  of  Demonstration  Scenario: 

Inferred  Belief  and  Affective  States  and  Predicted  Influence,  Compensatory 
Strategy,  and  GUI  /  DSS  Adaptation  (cont.) 


Time 

Frame 

Inferred 

Belief 

Affeetiv 
e  State 

Predicted 

Impact 

Selected 

Strategy 

GUI /DSS 
Adaptation 

..1 
oo  v 

Hostile 

contacts; 

Under  attack 

high 

Narrowed  focus  on 
HUD;  Miss 
incoming  "friendly 
ID" 

Redirect 
focus  to 
radar; 

Emphasize 
change  in 
status  or 
target 

Subject  Profile  #1: 

-  Display  string 
over  HUD  to  look 
at  radar 

-  Change  target  to 
friendly  icon, 
enlarge  size 

Subject  Profile  #2: 

-  Display  arrow 
over  HUD 
pointing  towards 
radar 

-  Change  target  to 
friendly  icon, 
display  string 
"Changed  status" 

9:  «<  30 

Hostile 

contacts; 

Under 

attack; 

Attacking 

high 

| 

The  remainder  of  this  section  shows  the  rules  used  to  derive  the  ABAIS  output  at  each  stage 
of  the  adaptation  process,  the  associated  data  driving  the  inferencing,  and  the  actual  GUI 
modifications,  for  each  frame  of  the  simulation. 


The  detailed  description  below  is  organized  to  follow  the  sequence  data  flow  through  the 
ABAIS  system  modules.  The  description  is  provided  from  the  point  of  view  of  the  lead  aircraft 
for  the  high-anxious  pilot  and  indicates  the  source  of  cues  when  these  are  arriving  from  other 
than  the  lead's  own  sensors.  First,  the  salient  cues  arriving  from  the  dynamic  task  environment 
are  listed,  followed  by  a  description  of  the  affect  and  belief  assessment,  showing  examples  of 
instantiated  rules  used  for  the  assessment.  Second,  the  rules  that  map  the  resulting  identified 
affect  and  belief  states  onto  their  predicted  effects  on  performance  are  listed.  Third,  the  rules 
that  map  the  predicted  effects  on  performance  onto  selected  strategy  are  listed.  Finally,  the  rules 
that  map  the  selected  strategy  onto  an  specific  adaptation  are  listed.  These  rules  are  combined 
with  the  pilot  information  preference  profile  to  arrive  at  the  final  set  of  GUI  modifications 
displayed  to  the  pilot.  The  resulting  cockpit  instrument  GUI  is  then  shown. 
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Recall  that  the  User  Assessment  rules  use  the  following  information  as  the  basis  for 
inferencing: 

•  Static  factors  provided  by  analyst  or  pilot  prior  to  run  and  calculated  at  the  beginning  of  the 

simulation  run  (task  context  factors,  personality  factors,  training  /  proficiency) 

•  Dynamic  factors  ,  generated  during  the  run  based  on  dynamic  task  environment  and  /  or 

information  known  about  pilot  (external  events  factors,  individual  history,  physiological 
signals) 

Rules  using  static  factors  are  instantiated  once,  at  the  beginning  of  the  task  script,  to 
determine  the  contribution  of  the  static  factors  to  the  overall  anxiety  level.  The  contributions  of 
these  rules  to  the  overall  anxiety  level  of  belief  state  remains  constant  throughout  a  given 
scenario.  Rules  using  dynamic  factors  are  instantiated  at  each  frame  of  the  task  script,  to 
determine  their  contribution  to  the  anxiety  weight  factor  during  that  frame.  The  static  rules  for 
this  scenario  are  shown  in  table  4.3-6.  The  dynamic  rules,  and  their  associated  triggering  cues, 
are  shown  in  the  respective  time  frames  in  the  detailed  simulation  description  below. 

Table  4.3-6:  Static  Factors  User  Assessment  Rules 


iTask  context  factors  i 

IF  (task  type  =  Detection) 

THEN  (anxiety  weight  factor  =  3) 

IF  (task  complexity  =  med) 

THEN  (anxiety  weight  factor  =  5) 

IF  (weather  =  clear) 

THEN  (anxiety  weight  factor  =  0) 

IF  (light  conditions  =  light ) 

THEN  (anxiety  weight  factor  =  0) 

Personality  factors 

IF  (emotional  stability  =  low) 

THEN  (anxiety  weight  factor  =  1 0) 

Training  /  Proficiency  factors 

IF  (air  combat  hours  =  low) 

THEN  (anxiety  weight  factor  =  6) 

IF  (air  combat  exercises  hours  =  low) 

THEN  (anxiety  weight  factor  =  7) 

IF  (training  hours  =  low) 

THEN  (anxiety  weight  factor  =  4) 

IF  (recency  of  air  combat  hours  =  low) 

THEN  (anxiety  weight  factor  =  9) 

IF  (recency  of  air  combat  exercises  hours 

=  low)  THEN  (anxiety  weight  factor  =  7) 

IF  (training  hours  =  low) 

THEN  (anxiety  weight  factor  =  6) 

The  critical  time  segments  of  the  scenario,  that  is,  those  demonstrating  the  ABAIS  GUI 
adaptations,  begin  at  frame  7,  which  occurs  at  30  nautical  miles.  During  this  segment  of  the 
scenario  the  wingman  datalink  provides  a  “Friendly  ID”  on  one  of  the  contacts  in  the  presumed- 
hostile  contact  group.  The  high-anxious  pilot  does  not  notice  this  information  and,  in  the  non- 
adapted  version,  deploys  the  weapon  against  the  presumed  hostile  aircraft.  In  the  adapted 
version,  this  fratricide  is  prevented  by  implementing  the  ABAIS-recommended  GUI  adaptations; 
namely,  emphasizing  the  IFF  return  and  directing  the  pilot’s  attention  to  the  radar  display,  away 
from  the  limited  HUD  display  and  from  readying  the  weapon. 
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The  discussion  below  describe  in  detail  the  ABAIS  performance  for  each  frame  during  a 
simulation  involving  a  high-anxious  pilot.  For  each  frame,  representative  screenshots  are  shown, 
displaying  the  incoming  cues,  subsets  of  the  instantiated  rule  sets  for  affect  and  belief 
assessment,  impact  prediction,  strategy  selection,  and  GUI  adaptation.  For  those  frames  where 
adaptations  are  generated,  both  the  adapted  and  the  non-adapted  pilot  GUI’s  are  shown, 
otherwise  only  one  pilot  GUI  is  shown.  Note  that  not  all  states  of  anxiety  induce  visible  GUI 
adaptations  in  the  discussion  below,  since  the  actual  display  of  the  GUI  /  DSS  adaptations  is 
controlled  by  the  adaptation  threshold,  whose  value  is  set  by  the  analyst  prior  to  each  run.  In 
other  words,  if  the  adaptation  threshold  is  set  of  ‘high’,  then  adaptations  will  only  be  displayed 
when  anxiety  level  reaches  the  ‘high’  value. 

Figure  4.3-1  shows  the  pilot’s  GUI  at  the  beginning  of  the  scenario  simulation.  (Since  at  this 
point  there  is  no  difference  between  the  adapted  and  the  non-adapted  GUI,  only  one  screenshot  is 
shown.) 


Figure  4.3-1:  Pilot’s  GUI  at  the  Start  of  the  Simulation  Run 
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The  remainder  of  this  section  describes  in  detail  the  simulation  run  implementing  Scenario 
#1.  Refer  to  tables  4.3-4  and  4.3-5  for  summaries  of  the  arriving  cues,  and  the  affect  and  belief 
states  of  the  low-  and  high-anxious  pilots. 

4.3.1  Scenario  1  -  Frame  1  - 100  nm 

Figure  4.3. 1-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  1.  These  cues,  along  with  pilot  background  information,  provide  the  data 
for  the  affect  and  belief  assessment  rules.  Figure  4.3. 1-2  shows  a  subset  of  the  affect  assessment 
rules  instantiated  during  frame  1.  Figure  4.3. 1-3  a  subset  of  the  belief  assessment  rules 
instantiated  during  frame  1.  Table  4.3. 1-1  shows  the  actual  environment  and  the  pilot’s  belief 
and  anxiety  level,  as  assessed  by  ABAIS.  Finally,  figure  4.3. 1-4  shows  the  pilot’s  GUI  during 
frame  1.  Again,  since  at  this  point  there  is  no  difference  between  the  adapted  and  non-adapted 
GUI,  only  one  screenshot  is  shown. 


Figure  4.3.1- 1:  Cues  Arriving  During  Frame  1  of  Simulation  Run 


Figure  4.3. 1-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During 

Frame  1  of  Simulation  Run 
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Figure  4.3. 1-3: 


Subset  of  Belief  Assessment  Rules  Instantiated  During 
Frame  1  of  Simulation  Run 


Table  4.3.1-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Anxiety  State 

Probable  hostile  contacts 

Unknown  contacts 

Low 
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Figure  4. 3. 1-4:  Pilot’s  GUI  at  Frame  1  of  Simulation  Run 

4.3.2  Scenario  1  -  Frame  2-  70nm 

Figure  4.3.2- 1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  2.  These  cues,  along  with  pilot  background  information,  provide  the  data 
for  the  affect  and  belief  assessment  rules.  Figure  4.3. 2-2  shows  a  subset  of  the  affect  assessment 
rules  instantiated  during  frame.  Figure  4.3. 2-3  shows  a  subset  of  the  belief  assessment  rules 
instantiated  during  this  frame.  Table  4.3 .2-1  shows  the  actual  environment  and  the  pilot’s  belief 
and  anxiety  level,  as  assessed  by  ABAIS.  Finally,  figure  4.3.2-4  shows  the  pilot’s  GUI  during 
frame  2.  Again,  since  at  this  point  there  is  no  difference  between  the  adapted  and  non-adapted 
GUI,  only  one  screenshot  is  shown. 
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Figure  4.3. 2-1 


Figure  4. 3. 2-2: 


:  Cues  Arriving  During  Frame  2  of  Simulation  Run 


Affect  Assessment 


■3 


IF  (heart rate  =  increasing) 

THEN(  anxiety  weight  factor  =  5) 

IF  ( heart  rate  variability  =  normal ) 
THEN(  anxiety  weight  factor  =  0) 

IF  (cleared to  fire  history  =  negative ) 
AND(  cleared  to  fire) 

THEN(  anxiety  weight  factor  =  8) 


Close 


subset  of  Affect  Assessment  Rules  Instantiated  During 
Frame  2  of  Simulation  Run 
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Belief  Assessment 


m 


IF  ( unknown  contacts  closing ) 
AND  (anxiety  level  How  ) 
THEN  (contact=  hostile) 


Close 


Figure  4. 3. 2-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During 

Frame  2  of  Simulation  Run 

Table  4. 3. 2-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Low-med 
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Figure  4. 3.2-4:  Pilot’s  GUI  at  Frame  2  of  Simulation  Run 

4.3.3  Scenario  1  -  Frame  3  -50  nm 

Figure  4.33-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  3.  These  cues,  along  with  pilot  background  information,  provide  the  data 
for  the  affect  and  belief  assessment  rules.  Figure  4.33-2  shows  a  subset  of  the  affect  assessment 
rules  instantiated  during  frame  3.  Figure  4.33-3  a  subset  of  the  belief  assessment  rules 
instantiated  during  frame  3.  Table  4.33-1  shows  the  actual  environment  and  the  pilot’s  belief 
and  anxiety  level,  as  assessed  by  ABAIS.  Finally,  figure  4.33-4  shows  the  pilot’s  GUI  during 
frame  3.  Again,  since  at  this  point  there  is  no  difference  between  the  adapted  and  non-adapted 
GUI,  only  one  screenshot  is  shown. 
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Figure  4.3.3- 1:  Cues  Arriving  During  Frame  3  of  Simulation  Run 


Figure  4.3.3-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During 

Frame  3  of  Simulation  Run 
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§25 Belief  Assessment 


IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  =  high ) 
THEN  ( contacts  hostile ) 

IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  >  low  ) 
THEN  ( contacts  hostile ) 

IF  ( unknown  type  from  NCTR ) 
AND  ( anxiety  level  >  low) 
THEN  ( contacts  hostile ) 


m 


MM 


ill 

Vrgg 

Si? 

£S>, 

t  C.'y  -y. 


if 


Figure  4. 3. 3-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During 

Frame  3  of  Simulation  Run 


Table  4.3.3-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot  s  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Low-med 
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Figure  4. 3. 3-4:  Pilot’s  GUI  at  Frame  3  of  Simulation  Run 

4.3.4  Scenario  1  -  Frame  4  -  40  nm 

Figure  4.3.4-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  4  of  the  simulation  run.  These  cues,  along  with  pilot  background 
information,  provide  the  data  for  the  affect  and  belief  assessment  rules.  Figure  43.4-2  shows  a 
subset  of  the  affect  assessment  rules  instantiated  during  frame  4  of  the  simulation  run,  and  figure 
43.4-3  shows  a  subset  of  the  belief  assessment  rules.  Table  43.1-1  shows  the  actual 
environment  and  the  pilot’s  belief  and  anxiety  level,  as  assessed  by  ABAIS.  Figure  43.2-4 
shows  a  subset  of  the  impact  prediction  rules  instantiated  during  this  frame,  and  figure  43.4-5 
shows  a  subset  of  the  strategy  selection  rules.  Finally,  figures  43.4-6  and  43.4-7  show  the 
pilot’s  GUI,  the  non-adapted  and  the  adapted  versions,  respectively. 
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Figure  4. 3. 4-1 


Figure  4. 3. 4-2: 


P5Cues 


mm 


jWACS  -  hostile 
NCTR  -  unknown 
IFF  -  unknown 
Active  radar-  3  contacts 
Cleared  to  fire  with  EID 
Contacts  -  multiple 


Close 


:  Cues  Arriving  During  Frame  4  of  Simulation  Run 


|=|  Affect  Assessment 


IF  ( heart  rate  =  increasing ) 

THEN(  anxiety  weight  factor  =  5) 

IF  ( heart  rate  variability  =  decreasing ) 
THEN(  anxiety  weight  factor  =  5) 

IF  ( cleared  to  fire  history  =  negative ) 
AND(  cleared  to  fire) 

THEN(  anxiety  weight  factor  =  8) 

IF  ( NCTR  output  =  unknown ) 

THEN(  anxiety  weight  factor  =  8) 

IF  ( number  of  contacts  increases ) 
THEN(  anxiety  weight  factor  =  6) 


“3 


■fain  GUI 

•ubset  of  Affect  Assessment  Rules  Instantiated  During 
Frame  4  of  Simulation  Run 
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.^Belief  Assessment 


IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  =  high ) 
m  THEN ( contact  =  hostile) 

IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  >  low  ) 
*  THEN  (contacts  hostile) 

IF  ( unknown  type  from  NCTR ) 
AND  (anxiety level  >  low) 
THEN  ( contacts  hostile ) 

IF  ( no  response  from  IFF ) 
AND  (anxiety level  >  low) 
THEN  ( contacts  hostile ) 


si 


-m 


H.  % 


Figure  4. 3. 4-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During 

Frame  4  of  Simulation  Run 


Impact  Prediction 


IF  (multiple  contacts  on  radar)  j| 

THEN  (attention  focused  on  signals  representing  unknowns  orthreats)  |H| 
aIF  (unknown  or  ambiguous  contacts  on  radar  |  NCTR  does  not  exist)  ill 
THEN  (assume  contacts  are  hostile) 


Si! 

Hi 


Figure  4.3. 4-4:  Subset  of  Impact  Prediction  Rules  Instantiated 
During  Frame  4  of  the  Simulation  Run 
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gH Strategy  Prediction 


I  IF  (contact =  unknown) 

ANDfcontact  belief=  hostile) 
THEN(emphasize  unknown  status) 
AND(collect  more  data) 


m 


Figure  4. 3. 4-5:  Subset  of  Strategy  Selection  Rules  Instantiated 
During  Frame  4  of  Simulation  Run 


Table  4.3.4- 1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Probable  hostile  contacts 

Med 
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Figure  4. 3. 4-6:  Pilot’s  GUI  at  Frame  4  of  Simulation  Run: 

Non-Adapted  Version 
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Figure  4. 3. 4-7:  Pilot’s  GUI  at  Frame  4  of  Simulation  Run:  Adapted  Version 

4.3.5  Scenario  1  -  Frame  5  -  37nm 

ABAIS  inferencing  and  GUI  remain  largely  unchanged  during  this  frame.  The  pilot’s  anxiety 
level  remains  at  ‘medium’,  his  belief  state  remains  at  ‘hostile  contacts’;  in  other  words,  the  pilot 
is  interpreting  the  unknown  probable  hostile  contacts  as  in  fact  hostile.  Table  4.3 .5-1  shows  the 
actual  environment  and  the  pilot’s  belief  and  anxiety  level,  as  assessed  by  ABAIS.  The  cockpit 
instruments  remain  unchanged  from  those  shown  in  frame  4  above. 


Table  4. 3. 5-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Med 

114 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


4.3.6  Scenario  1  -  Frame  6  -  35  nm 

ABAIS  inferencing  and  GUI  remain  largely  unchanged  during  this  frame.  The  pilot’s  anxiety 
level  remains  at  ‘medium’,  his  belief  state  remains  at  ‘hostile  contacts’;  in  other  words,  the  pilot 
is  interpreting  the  unknown  probable  hostile  contacts  as  in  fact  hostile.  Table  43.6-1  shows  the 
actual  environment  and  the  pilot’s  belief  and  anxiety  level,  as  assessed  by  ABAIS.  The  cockpit 
instruments  remain  unchanged  from  those  shown  in  frame  4  above. 


Table  4. 3. 6-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Med 

4.3.7  Scenario  1  -  Frame  7  -  30  nm 


Figure  43.7-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  7.  These  cues,  along  with  pilot  background  information,  provide  the  data 
for  the  affect  and  belief  assessment  rules.  Figure  43.7-2  shows  a  subset  of  the  affect  assessment 
rules  instantiated  during  frame  7.  Figure  43.7-3  a  subset  of  the  belief  assessment  rules 
instantiated  during  frame  7.  Table  43.7-1  shows  the  actual  environment  and  the  pilot’s  belief 
and  anxiety  level,  as  assessed  by  ABAIS.  Figure  43.7-4  shows  a  subset  of  the  strategy 
suggestion  rules  instantiated  during  this  frame.  At  this  stage  the  pilot’s  attention  is  focused  on 
the  alert  window,  informing  him  of  a  radar  lock.  The  adaptation  rules  direct  ABAIS  to 
emphasize  the  unknown  status  of  the  IFF,  to  prevent  possible  task  neglect.  Figures  43.7-5  and 
43.7-6  show  the  pilot’s  GUI,  the  non-adapted  and  the  adapted  versions,  respectively. 


Cues 


m 

s# 

•H 


AWACS- hostile 
NCTR  -  unknown 
IFF  -  unknown 
Active  radar-  3  contacts 
RWR  -  unknown  air-to-air 
RWR-^AM  lock 
Cleared  to  fire  with  EID 
Contacts  -  multiple 


Figure  4. 3. 7-1:  Cues  Arriving  During  Frame  7  of  Simulation  Run 
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F  ( heart  rate  =  increasing ) 

THEN(  anxiety  weight  factor  =  5 ) 

7  ( heart  rate  variability  =  decreasing ) 
THEN(  anxiety  weight  factor=  5 ) 

7  ( cleared  to  fire  history  =  negative ) 
AND(  cleared  to  fire ) 

THEN(  anxiety  weight  factors  8 ) 

7  ( NCTR  outputs  unknown ) 

THEN(  anxiety  weight  factorjs  8 ) 

7  ( RWR  outputs  air-to-air  unknown ) 
THEN(  anxiety  weight  factor  s  8 ) 

7  (number  of  contacts  increases) 
THEN(  anxiety  weight  factor  =  6 ) 


Figure  4.3.7-2:  Subset  of  Affect  Assessment  Rules  Instantiated  During 

Frame  7  of  Simulation  Run 


H| Belief  Assessment 


[F  (unknown  contacts  closing) 
AND  ( anxiety  level  =  high ) 
THEN  ( contacts  hostile ) 

IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  >  low  ) 
THEN  ( contacts  hostile ) 

IF  ( unknown  type  from  NCTR ) 
AND  ( anxiety  level  >  low) 
THEN  ( contacts  hostile ) 

IF  ( no  response  from  IFF ) 

AND  ( anxiety  level  >  low) 
THEN  ( contacts  hostile ) 

IF  ( unknown  radar  lock  on  RWR ) 
AND  ( anxiety  level  >  low) 
THEN  ( aircraft  under  attack) 


Figure  4.3. 7-3:  Subset  of  Belief  Assessment  Rules  Instantiated  During 

Frame  7  of  Simulation  Run 
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||| Strategy  Prediction 


m 


[F  (contact  =  unknown) 

AND(contact  belief  =  hostile) 

THEN(emphasize  unknown  status) 

AND(collect  more  data) 

IF(likelihood  of  premature  attacks  high) 
THEN(display  all  available  info  about  enemy  a/c) 
AND(enhance  display  of  enemy  a/c  info) 


Close 


Figure  4. 3. 7-4  Subset  of  Strategy  Suggestion  Rules  Instantiated 
During  Frame  7  of  Simulation  Run 


Table  4. 3. 7-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Under  attack 

Med-high 
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Sifiiuiditon  wtfhuut  Adaptation 


Figure  4. 3. 7-5:  Pilot’s  GUI  at  Frame  7  of  Simulation  Run: 

Non-Adapted  Version 
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Figure  4. 3. 7-6:  Pilot’s  GUI  at  Frame  7  of  Simulation  Run:  Adapted  Version 

4.3.8  Scenario  1  -  Frame  8  <  30  nm 

Figure  4.3.8-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  8.  Figure  43.8-2  shows  the  actions  being  performed  by  the  pilot  during 
this  frame.  Figure  43.8-3  a  subset  of  the  belief  assessment  rules  instantiated  during  frame  8. 
Figure  43.8-4  shows  a  subset  of  the  impact  prediction  rules  instantiated  during  this  frame  and 
figure  43.8-5  shows  a  subset  of  the  strategy  selection  rules  instantiated.  Figure  43.8-6  shows  a 
subset  of  the  GUI  Adaptation  rules  instantiated  during  this  frame.  At  this  point  during  the 
scenario  the  pilot’s  attention  is  focused  on  the  HUD,  where  he  is  ‘centering  the  dot’  in  response 
to  his  belief  that  he  is  under  attack  from  a  closing  hostile  aircraft.  The  adaptation  rules  suggest 
that  the  pilot  direct  his  attention  to  the  radar  display,  and  suggest  that  the  system  emphasize  the 
unknown  status  of  the  IFF  to  prevent  possible  fratricide.  Figures  43.8-7  and  43.8-8  show  the 
pilot’s  GUI,  the  non-adapted  and  the  adapted  versions,  respectively. 
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;  AWACS-  hostile 
NCTR-  unknown 
IFF -friendly 
Active  radar-  3  contacts 
RWR  -  unknown  air-to-air 
Wing  RWR  -  unknown  air-t 
RWR  -  SAM  lock 
Cleared  to  fire  -  given 


Figure  4.3.8-1:  Cues  Arriving  During  Frame  8  of  Simulation  Run 


111!  Actions 

E3 

Initiate  intercept 
Attempt  IFF 
.  Acquiring  target. 

Focus  on  HUD 

“13 : 

11 

-J' 

Close 

j 

Figure  4. 3. 8-2:  Pilot’s  Actions  During  Frame  8  of  the  Simulation  Run 


I  ^  B  elief  Assessment 


IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  =  high ) 
THEN  ( contact  =  hostile ) 

IF  ( unknown  contacts  closing ) 
AND  (cleared  to  fire  ) 

THEN  ( contacts  hostile ) 

IF  ( unknown  contacts  closing ) 
AND  ( anxiety  level  >  low  ) 
THEN  ( contact  =  hostile ) 

IF  ( unknown  type  from  NCTR ) 
AND  ( anxiety  level  >  low) 
THEN  (contact  =  hostile) 

IF  ( unknown  radar  lock  on  RWR ) 
AND  ( anxiety  level  >  low) 
THEN  (aircraft  under  attack) 


Figure  4. 3. 8-3: 


Subset  of  Belief  Assessment  Rules  Instantiated  During 
Frame  8  of  Simulation  Run  & 
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Impact  Prediction 


IF  (unknown  air-to-air  radar  lock  on  RWR)  3 

THEN  (assume  under  attack) 

4  IF  (no  reply  from  IFF) 

-v  THEN  (assume  contacts  are  hostile) 
i  IF  (anxiety  >  med) 

'  AND  (contact  assumed  hostile) 

|  AND  (no  IFF)) 

THEN  (likelihood  of  premature  attack=  high) 


Close 


Figure  4. 3. 8-4:  Subset  of  Impact  Prediction  Rules  Instantiated 
During  Frame  8  of  the  Simulation  Run 


|H^ Strategy  Prediction 


IF  (recent  change  in  radar  target  states) 
THEN(emphasize  change  in  status  of  return) 

IF  (attention  focus  =  HUD) 

AND(incoming  radar  data) 

THEN(redirect focus  to  radar) 

IFflikellhood  of  premature  attack  =  high) 
THEN(display  all  available  info  about  enemy  a/c) 
AND(enhance  display  of  enemy  a/c  info) 


Close 


Figure  4. 3. 8-5:  Subset  of  Strategy  Selection  Rules  Instantiated 
During  Frame  8  of  Simulation  Run 


H|GU1  Adaptation 


m 


[F  (goal  =  redirect  attention  to  radar) 

THEN(implement  attention  capture  preference  according  to  subjf 
IF  (goal  =  enhance  target  status) 

THEN(highlight  target  information  according  to  subject  profile  pre 


Close 


Figure  4. 3. 8-6:  Subset  of  GUI  Adaptation  Rules  Instantiated 
During  Frame  8  of  Simulation  Run 
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Table  4.3.8-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth  I  Pilot's  Belief  State 
_ _  (Situation  Assessment) 


Pilot's  Affective  State 


Figure  4. 3. 8-7 :  Pilot’s  GUI  at  Frame  8  of  Simulation  Run: 

Non-Adapted  Version 
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Figure  4. 3. 8-8:  Pilot’s  GUI  at  Frame  8  of  Simulation  Run:  Adapted  Version 

4.3.9  Scenario  1  -  Frame  9-  «  30  nm 

Figure  4.3.9-1  shows  a  screenshot  of  the  ABAIS  cue  monitoring  window,  showing  the  cues 
arriving  during  frame  9.  Figure  43.9-2  shows  a  subset  of  the  impact  prediction  rules  instantiated 
during  this  frame  and  figure  43.9-3  shows  a  subset  of  the  strategy  selection  rules  instantiated. 
Figure  43.9-4  shows  a  subset  of  the  GUI  Adaptation  rules  instantiated  during  this  frame.  At  this 
point  during  the  scenario  the  pilot’s  attention  is  focused  on  the  HUD,  where  he  is  ‘centering  the 
dot’  in  response  to  his  belief  that  he  is  under  attack  from  a  closing  hostile  aircraft.  The  adaptation 
rules  suggest  that  the  pilot  should  direct  his  attention  to  the  radar  display,  and  suggest  that  the 
system  emphasize  the  unknown  status  of  the  IFF  to  prevent  possible  fratricide.  Figures  43.9-5 
and  43.9-6  show  the  pilot’s  GUI,  the  non-adapted  and  the  adapted  versions,  respectively. 
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M  Cues 


AWACS- hostile 
NCTR  -  unknown 
IFF  -  friendly 
| Active  radar  -  3  contacts 
RWR-  unknown  air-to-air 
jWing  RWR  -  unknown  air-to-air 
RWR  -  SAM  lock 
Cleared  to  fire  -  given 
Contacts  -  multiple 
Wing  -  Cease  fire! 


* 


Close 


Figure  4. 3. 9-1:  Cues  Arriving  During  Frame  9  of  Simulation  Run 


Impact  Prediction 


IF  (unknown  air-to-air  radar  lock  on  RWR) 
THEN  (assume  under  attack) 

IF  (no  reply  from  IFF) 

THEN  (assume  contacts  are  hostile) 

IF  (anxiety  >  med) 

AND  (contact  assumed  hostile) 

AND  (no  IFF) ) 

THEN  (likelihood  of  premature  attack  =  high) 


>1 


Close 


Figure  4.3.9-2:  Subset  of  Impact  Prediction  Rules  Instantiated 
During  Frame  9  of  the  Simulation  Run 


Strategy  Prediction 


IF  (recent  change  in  radar  target  status) 
THEN(emphasize  change  in  status  of  return) 
!F(likelihood  of  premature  attack  =  high) 
THEN(display  all  available  info  about  enemy  a/c) 
AND(enhance  display  of  enemy  a/c  info) 


Close 


J 


Figure  4.3.9-3:  Subset  of  Strategy  Selection  Rules  Instantiated 
During  Frame  9  of  Simulation  Run 
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m 


GUI  Adaptation 


m 


IF  (goal  =  redirect  attention  to  radar) 

THEN(implement  attention  capture  preference  according  to  subjs 
IF  (goal  =  enhance  target  status) 

THEN(highlight  target  information  according  to  subject  profile  pre 


Close 


Figure  4. 3. 9-4:  Subset  of  GUI  Adaptation  Rules  Instantiated 
During  Frame  9  of  Simulation  Run 


Table  4. 3. 9-1:  Scenario  Environment  and  Pilot’s  Belief  and  Anxiety  State 


Actual  Ground  Truth 

Pilot's  Belief  State 
(Situation  Assessment) 

Pilot's  Affective  State 

Probable  hostile  contacts 

Hostile  contacts 

Under  attack 

High 

Rules  deriving  belief  state: 

IF  (unknown  radar  lock  on  RWR)  AND  (anxiety  level  =  >  low)  THEN  (contact  =  hostile) 

IF  (unknown  radar  lock  on  RWR)  AND  (anxiety  level  =  >  low)  THEN  (aircraft  under  attack) 
IF  (cleared  to  fire)  AND  (unknown  radar  lock  on  RWR) _ THEN  (aircraft  under  attack) 


Rules  deriving  affective  state: 


Individual  history  factors 

IF  (cleared  to  fire) 

THEN  (anxiety  weight  factor  =  8) 

External  event  factors 

IF  (unknown  radar  lock  on  RWR) 

THEN  (anxiety  weight  factor  =  8) 

IF  (wing  unknown  radar  lock  on  RWR) 

THEN  (anxiety  weight  factor  =  8) 

Physiological  factors 

IF  (heart  rate  =  increasing) 

THEN  (AWF  =  5) 
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Figure  4.3 .9-5:  Pilot’s  GUI  at  Frame  9  of  Simulation  Run: 

Non-Adapted  Version 
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Figure  4. 3. 9-6:  Pilot’s  GUI  at  Frame  9  of  Simulation  Run:  Adapted  Version 
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5.  Summary,  Conclusions,  and  Recommendations 

This  chapter  summarizes  the  key  tasks  conducted  under  this  effort  and  presents  the  major 
conclusions. 

5.1  Summary 

The  approach  taken  under  this  effort  focused  on  the  proof-of-concept  demonstration  of  an 
Affect  and  Belief  Adaptive  Interface  System  (ABAIS).  ABAIS  assists  the  user  (e.g.,  pilot)  in  the 
detection  of,  and  adaptation  to,  user  affective  states  (e.g.,  anxiety),  dominant  personality 
characteristics  (e.g.,  obsessiveness),  and  current  set  of  beliefs  that  may  influence  performance 
(activated  knowledge  schemata)  (e.g.,  pilot’s  belief  that  he  is  under  attack  by  hostile  aircraft),  in 
an  attempt  to  counter  the  associated  performance  biases.  Seven  specific  tasks  comprised  our 
effort: 

•  Defining  a  scenario  for  ABAIS  demonstration 

•  Identifying  generic  effects  of  affective  states  and  decision-making  biases  resulting  from 
intuitive,  idiosyncratic  beliefs  from  existing  research  literature 

•  Identifying  specific  likely  effects  of  affective  states  and  examples  of  individual, 
idiosyncratic  beliefs  that  might  influence  performance  from  knowledge  elicitation 
interviews  with  expert  Air  Force  pilots  and  research  personnel 

•  Developing  knowledge  bases  for  each  of  the  four  modules  comprising  the  ABAIS  system 
architecture 

•  Designing  and  implementing  a  prototype  ABAIS  architecture 

•  Demonstrating  the  proof-of-concept  of  the  ABAIS  framework  and  prototype  in  the  context 
of  selected  test  scenarios 

•  Defining  requirements  for  Phase  II  development  and  commercialization 

The  activities  performed  under  each  of  these  tasks  are  summarized  below. 

We  first  defined  a  scenario  for  ABAIS  demonstration  through  a  series  of 
knowledge  elicitation  interviews  with  subject  matter  experts  (SME’s)  (USAF  and  Navy  fighter 
pilots  and  psychologists).  After  considering  several  options  of  tasks  and  task  contexts  (e.g., 
instrument  failures  and  intercept  geometries)  to  induce  anxiety,  we  selected  the  sweep  task  and  a 
series  of  BVR  air-to-air  combat  encounter  events  to  demonstrate  the  ABAIS  adaptive 
methodology  in  a  real-time,  dynamic  context. 

We  then  identified  generic  effects  of  affect-  and  belief-induced 
performance  biases  by  reviewing  empirical  literature  in  four  areas:  1)  experimental, 
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cognitive,  and  applied  psychology;  2)  personality  theory;  3)  human  factors;  and  4)  cognitive 
engineering.  The  findings  were  then  evaluated  for  their  potential  in  providing  information  about 
the  effects  of  affect-,  personality  trait-,  and  belief-induced  biases  on  performance. 

We  then  identified  specific  likely  effects  of  affective  states,  selected 
personality  factors,  and  individual  beliefs  on  flight  performance  in  the  context 
of  the  demonstration  scenario  described  above.  Given  the  relative  paucity  of  empirical  studies  in 
this  area,  the  primary  means  of  identifying  these  effects  were  knowledge  elicitation  interviews 
with  pilots  and  other  aviation  subject  matter  experts  (SME's). 

We  then  developed  knowledge  bases  for  each  of  the  four  modules  comprising  the 
ABAIS  system  from  the  knowledge  derived  both  from  the  literature  search,  and  from  knowledge 
elicitation  interviews  with  SME's.  For  the  User  State  Assessment ,  this  effort  consisted  first  of 
identifying  the  multiple  categories  of  factors  necessary  to  infer  the  user's  affective  state, 
personality  trait,  and  predominant  beliefs,  and  second,  of  constructing  the  rules  mapping  these 
factors  onto  the  space  of  possible  affective  and  belief  states.  For  the  Impact  Prediction  module, 
this  effort  consisted  of  constructing  rules  that  mapped  the  identified  affective  state,  personality 
trait,  or  active  belief  onto  the  most  likely  effects  on  various  elements  of  the  sweep  task 
performance.  For  the  Strategy  Selection  module,  we  developed  rules  mapping  the  specific 
performance  bias  (e.g.,  task  neglect,  cue  misinterpretation,  checking  behavior)  onto  a 
corresponding  series  of  compensatory  strategies  counteracting  these  biases  (e.g.,  redirecting 
focus  onto  neglected  data,  reminders  of  increasing  vulnerability  due  to  a  delayed  decision). 
Finally,  for  the  GUI  /  DSS  Adaptation  module,  we  developed  rules  implementing  specific 
modifications  of  the  pilot  cockpit  instrument  displays  corresponding  to  the  suggested  strategy 
(e.g.,  highlighting  information  to  capture  pilot's  attention,  displaying  additional  information  to 
counter  predicted  biases,  etc.). 

We  designed,  constructed,  integrated,  and  evaluated  a  wireless  heart  rate 
monitor.  The  monitor  consists  of  a  light-weight  belt  worn  around  the  torso  and  an  associated 
wireless  transmitter.  In  our  initial  demonstration  we  integrated  the  monitor  with  simple  ABAIS 
GUI  adaptations  to  demonstrate  ABAIS'  capability  to  detect  and  respond  to  changing 
physiological  signals. 

We  then  demonstrated  the  proof-of-concept  of  the  ABAIS  framework  and 

prototype  in  the  context  of  selected  task  scenarios.  The  distinct  pilot  profiles  generated  different 
affective  and  belief  states  at  different  points  of  the  scenario.  The  resulting  GUI  /  DSS 
adaptations  were  derived  via  the  ABAIS  adaptive  methodology,  using  the  rules  in  the  four 
ABAIS  modules.  The  output  of  the  GUI  /  DSS  Adaptation  recommendations  was  then  shown  on 
the  corresponding  cockpit  instruments. 


129 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Finally,  we  defined  requirements  for  Phase  II  full-scope  development  and 
commercialization.  We  focused  on  identifying  further  development  and  demonstration 
requirements  to  be  met  for  a  full-scope  affective  adaptive  user  interface  for  a  variety  of  real-time, 
dynamic  environments  (e.g.,  AWACS  tasks).  We  focused  on  environments  requiring  team 
coordination,  as  these  are  becoming  increasingly  common  and  provide  a  rich  context  in  which  to 
demonstrate  the  proposed  ABAIS  methodology.  To  define  commercialization  requirements,  we 
identified  promising  commercial  market  areas  and  particular  market  segments  that  could  benefit 
from  the  development  of  a  suitably  specialized  affective,  adaptive  interface  and  decision  support 
tool. 


5.2  Conclusions 

The  primary  result  of  this  study  was  a  proof-of-concept  demonstration  of  an  Affect  and 
Belief  Adaptive  Interface  System  (ABAIS),  designed  to  provide  individualized  GUI  and  DSS 
adaptations  based  on  the  user's  affective  and  belief  state.  Specific  results  are  outlined  below, 
first  for  the  effort  as  a  whole,  and  then  organized  by  the  individual  subtasks  conducted. 

Overall  ABAIS  Adaptive  Methodology  Framework  Development  of  the  Phase  I 
ABAIS  proof-of-concept  demonstrated  feasibility  of  the  overall  adaptive  methodology  and  its 
implementation.  The  four  modules  comprising  the  ABAIS  architecture  were  able  to  assess  the 
user  state  using  a  knowledge-based  approach  and  information  from  a  variety  of  sources  (e.g., 
static  task  context,  dynamic  external  events  occurring  during  the  task,  pilot's  individual  history, 
pilot's  personality  traits,  pilot's  training  and  proficiency,  and  simulated  physiological  data), 
predict  the  effects  of  this  state  in  the  context  of  the  demonstration  task,  and  suggest  and 
implement  specific  GUI  adaptation  strategies  (e.g.,  modify  an  icon  or  display  to  enhance 
visibility),  based  on  the  pilot's  individual  information  presentation  preferences. 

Results  of  this  demonstration  indicate  the  feasibility  of  the  approach  as  a  whole,  and  of 
knowledge-based  affect  assessment  and  GUI  adaptation.  The  implementation  allows  the 
specification  of  highly-individualized  baseline  pilot  profiles,  which  can  be  used  to  infer 
individual  reactions  to  a  variety  of  specific  events,  based  on  the  pilot's  background,  personality, 
and  individual  history  (i.e.,  specific  events  in  recent  past),  and  selected  physiological  data.  A  key 
finding  was  the  demonstration  of  the  feasibility  of  using  available  electronic  data  about  the  task 
environment  (e.g.,  radar  contacts)  as  basis  for  both  affect  and  belief  assessment 

Specifically,  we  believe  that  the  following  factors  represent  the  key  requirements  for  the 
development  of  an  adaptive,  affective  interface: 

•  Availability  of  highly  individualized  data  about  the  system  user,  including  details  of  past 
performance,  individual  history,  personality  traits,  and  physiological  data; 
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•  Availability  of  a  wide  variety  of  task-specific  data  in  an  electronic  format; 

•  Use  of  multiple  methods  and  multiple  sources  of  data  for  assessing  the  user's  affective  and 

belief  state; 

•  Ability  to  fine-tune  the  rule-bases  and  inferencing  to  "personalize"  the  system  to  the 

individual  user-task  context. 

The  specific  findings  supporting  these  conclusions,  and  the  detailed  findings  under  each  of 
the  seven  tasks  comprising  the  Phase  I  effort,  are  summarized  in  detail  in  section  1.5. 

5.3  Recommendations 

On  the  basis  of  these  Phase  I  results,  a  Phase  II  effort  is  recommended  which  focuses  on  the 
design,  development,  and  evaluation  of  a  full-scope  Affect  and  Belief  Adaptive  Interface 
(ABAIS)  system  and  testbed  environment.  The  overall  functionality  of  the  system,  and  its  key 
features,  are  briefly  outlined  below. 

Core  ABAIS  Architecture  The  core  ABAIS  architecture  implements  an  adaptive 
methodology,  summarized  in  table  3.1-1,  and  consists  of  four  modules:  1)  user  state  assessment, 
which  identifies  the  user's  affective  state  and  task-relevant  beliefs;  2)  impact  prediction,  which 
identifies  effects  of  the  user  state  on  performance;  3)  strategy  selection ,  which  selects  a 
compensatory  strategy;  and  4)  GUI  adaptation,  which  modifies  the  user  interface  content  and 
format  to  enhance  detection,  recognition,  and  assimilation  (i.e.,  to  improve  situation  awareness). 
The  ABAIS  architecture  implementing  this  methodology,  in  the  context  of  the  Phase  I 
demonstration  environment,  is  shown  in  figure  3.1-1.  Under  Phase  I  we  successfully  prototyped 
both  the  methodology  and  the  architecture.  Under  Phase  II,  we  would  build  upon  the  existing 
ABAIS  architecture  by  providing  the  following:  1)  additional  modules  to  handle  explicit  task 
parameter  and  knowledge-base  modification;  2)  enhanced  connectivity  among  individual  ABAIS 
modules  to  provide  feedback  to  the  user  assessment  module,  thereby  enhancing  the  assessment 
process;  and  3)  instrumentation  of  ABAIS  with  the  necessary  features  to  function  in  a  team 
setting  (i.e.,  ability  to  instantiate  multiple  ABAIS  copies  on  separate  workstations  and 
communication  among  these,  multiple  user  assessment,  etc.).  The  implementation  of  ABAIS 
within  a  team  context  would  also  require  the  definition  and  maintenance  of  multiple  ABAIS 
user  profiles,  and  support  for  real-time  interaction  among  these,  to  provide  the  basis  of 
coordinating  adaptation  across  multiple  team  members. 

ABAIS  Run-Time  Execution  Modes  Under  Phase  /,  the  core  ABAIS  framework  was 
integrated  within  a  dynamic  flight  simulation  environment  and  supported  two  modes  of  system 
operation:  1)  pilot-as-user  mode,  where  the  user  actually  flies  the  aircraft  and  interacts  with  a 
simulated  environment  consisting  of  other  friendly  aircraft,  enemy  aircraft,  and  missiles;  and  2) 
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analyst-as-user,  where  the  analyst  watches  a  scripted  task  and  monitors  the  scripted  pilot's 
performance,  as  well  as  the  rule-based  inferencing  results.  To  support  these  two  modes  of 
operation,  ABAIS  was  integrated  with  a  simulation  environment  supporting  both  human-in-the- 
loop  flight  simulation,  and  script-based  agent  simulation.  The  script  definition  functionality 
allowed  the  interactive  graphical  specification  of  customized  scripts  within  the  general  sweep 
task  context,  consisting  of  varying  number  of  aircraft  with  associated  autonomous  behaviors,  via 
a  variety  of  pop-up  menus  associated  with  the  individual  cells. 

Under  Phase  II,  we  plan  to  maintain  the  two-mode  operation  to  provide  maximum  flexibility 
in  the  testing  and  evaluation  of  a  full-scope  ABAIS  within  a  team  context.  However,  the  focus 
will  be  on  the  full-scope  development  and  support  of  a  human-in-the-loop  mode,  to  provide 
opportunities  for  closed-loop  assessment  and  adaptation  in  actual  dynamic  team  tasks. 

User  Profiles  Under  Phase  I,  profiles  of  representative  users  (e.g.,  anxiety-tolerant  and 
anxiety-prone  pilots)  were  constructed  and  used  as  basis  for  performing  the  dynamic  affect 
assessment  during  the  demonstration  sweep  task.  Under  Phase  II,  we  would  still  support  the 
construction  of  profiles  for  scripted  task  executions,  but  the  emphasis  will  be  on  the  real-time 
assessment  of  a  variety  of  dynamic  factors,  including  diagnostic  task  data  (e.g.,  time  and 
accuracy  measures  of  performance),  self-assessment  reports  (e.g.,  indication  whether  or  not  the 
user  feels  anxious  at  a  particular  stage  of  the  task),  and  physiological  measures  (i.e.,  heart  rate). 

ABAIS  Demonstration  Task  Under  Phase  I,  we  demonstrated  ABAIS  functionality  in 
the  context  of  a  sweep  task,  focusing  on  adaptation  with  respect  to  a  single  user  (i.e.,  a  single 
pilot).  The  primary  ABAIS  demonstration  used  the  analyst-as-user  mode,  due  to  the  precise 
coordination  required  to  "fly"  the  sweep  task  demonstration  script  and  simulate  the  desired 
sequence  of  events  to  demonstrate  real-time  adaptation.  In  this  mode  the  aircraft  is  controlled  by 
a  script  that  simulates  the  pilot  behavior  (e.g.,  “centering  the  dot”  on  HUD)  and  the  dynamic  task 
environment  (e.g.,  changed  contact  status  due  to  incoming  IFF  data).  The  pilot-as-user  (human- 
in-the-loop)  mode  was  demonstrated  separately,  during  an  evaluation  of  physiological  user  state 
assessment.  This  evaluation  involved  the  design  and  development  of  a  wearable  heart  rate 
sensor,  whose  output  is  linked  directly  into  the  ABAIS  GUI  /  DSS  Adaptation  module  to  provide 
modified  GUI’s  in  response  to  specific  changes  in  the  user’s  heart  rate.  The  successful 
implementation  of  this  sensor  demonstrated  the  feasibility  of  using  real-time  physiological  data 
to  drive  the  ABAIS  adaptation  logic. 

Under  Phase  II,  the  scope  of  demonstration  will  be  extended  to  a  synthetic  team  task 
environment,  the  Distributed  Dynamic  Decision-Making  (DDD).  This  environment,  developed 
by  Aptima,  Inc.,  has  been  successfully  used  for  a  variety  of  empirical  evaluations  of  human-in- 
the-loop  design  issues.  As  such,  it  provides  a  flexible  tool  for  the  exploration  of  ABAIS 
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adaptations  across  multiple  team  tasks.  As  mentioned  above,  this  extension  will  involve  the 
capability  to  instantiate  multiple  “copies”  of  ABAIS  across  separate  workstations,  the  ability  to 
specify  multiple  profiles  and  assess  of  multiple  user  states,  and  the  ability  to  provide  the 
necessary  communication  mechanisms  to  ensure  adaptation  to  multiple  users.  The  team  task  was 
selected  based  on  the  increasing  importance  of  team  environments  in  the  USAF,  and  recent 
results  indicating  critical  affect-induced  team  effects  and  biases.  Specifically,  several  recent 
studies  focus  on  stress  related  decision  making  deficits  which  include  psycho-social  factors  (i.e., 
intra-crew  tensions).  The  impact  on  decision  was  found  to  be  most  extreme  in  complex  decisions 
with  high  consequence.  Both  perceptual  narrowing  and  failure  to  note  disconfirming  data  were 
observed  (Prince  et  al.,  1994;  Driskell  and  Salas,  1991). 

Individual  vs.  Team  Adaptations  Under  Phase  /,  individual  adaptations  were 
implemented  and  explored  in  the  context  of  the  sweep  task.  Specifically,  selected  cockpit 
instruments  (e.g.,  HUD,  radar)  were  simulated  and  adaptations  were  implemented  within  these 
displays  by  enhancing  icons  or  redirecting  attention  to  critical  incoming  data.  Under  Phase  II, 
we  would:  1)  continue  individual  adaptations  but  would  implement  ABAIS  in  the  context  of  a 
distributed  team  task  environment;  2)  would  enhance  the  level  of  adaptation;  3)  would  explore  a 
variety  of  specific  applications  of  the  user  state  assessment. 

The  ABAIS  Testbed  Environment  A  key  objective  of  the  overall  effort  is  to  develop 
a  flexible  testbed  for  the  exploration  of  multiple  affective  assessment  methods  and  adaptation 
strategies.  Under  Phase  I,  we  developed  an  analyst  GUI  supporting  the  flexible  specification  of 
background  pilot  information,  selection  of  run  time  parameters,  and  system  performance 
monitoring.  The  analyst  displays  include  a  variety  of  windows  showing  dialog  boxes  for  defining 
the  background  information  necessary  for  the  knowledge-based  user  assessment  (e.g.,  pilot 
preference  profile,  pilot  individual  history);  dialog  boxes  for  specifying  the  system  run  time 
parameters  (e.g.,  what  windows  should  be  visible  for  monitoring  system  performance);  and 
windows  allowing  monitoring  of  system  performance  (e.g.,  rule  base  and  rule  instantiation, 
physiological  data,  inferred  pilot  affective  state). 

Under  Phase  II,  we  would  build  upon  this  existing  functionality  to  enhance  the  analyst  GUI 
and  interaction  facilities.  Specifically,  we  would  include  a  capability  for  dynamical  monitoring 
of,  and  interaction  with,  the  knowledge-based  inferencing,  and  will  provide  a  means  of  flexibly 
combining  the  complementary  affect  assessment  methods  and  adaptation  strategies  across 
multiple  task  scenarios.  We  would  integrate  these  facilities  with  the  existing  DDD  software 
environment,  to  provide  support  for  exploration  of  ABAIS  across  multiple  team  tasks  and  team 
configurations. 


133 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


6.  References 

Ackerman,  P.L.,  Sternberg,  R.J.  and  Glaser,  R.  (1989).  Learning  and  individual  differences.  New 
York:  W.H.Freeman. 

Araujo,  A.F.R.  (1993).  Emotions  Influencing  Cognition  In  WAUME  '93:  Workshop  on 
Architectures  Underlying  Motivation  and  Emotion.  Birmingham,  UK:  The  University  of 
Birmingham. 

Badre,  A.  N.  (1978).  Selecting  and  Representing  Information  Structures  for  Battlefield  Decision 
Systems  ARI  Technical  Report  79-A20. 

Bates,  J.,  Loyall,  A.B.,  and  Reilly,  W.S.  (1992).  Integrating  Reactivity,  Goals,  and  Emotion  in  a 
Broad  Agent  In  Proceedings  of  the  14th  Meeting  of  the  Cognitive  Science  Society. 

Berry,  D.  (1987).  The  problem  of  implicit  knowledge.  Expert  Systems,  4, 144-151. 

Blaney,  P.H.  (1986).  Affect  and  Memory.  Psychological  Bulletin,  99  (2),  229-246. 

Bower,  G.H.  (1981).  Mood  and  Memory.  American  Psychologist,  36, 129-148. 

Briggs,  K.C.  and  Myers,  I.B.  (1977).  Myers-Briggs  Type  Indicator.  Palo  Alto,  CA:  Consulting 
Psychologists  Press,  Inc. 

Broadbent,  D.E.,  Broadbent,  M.H.P,  and  Jones,  J.L.  (1986)  Performance  correlates  of  self- 
reported  cognitive  failure  and  obsessionality.  British  Jnl.  of  Clin.  Psych.,  25, 285-299. 

Cacioppo,  J.T.,  Klein,  D.J.,  Bemston,  G.G.,  and  Hatfield,  E.  (1993).  The  Psychophysiology  of 
Emotion.  In  Handbook  of  Emotions,  M.  Lewis  and  J.  Haviland,  eds.  New  York:  Guilford. 

Callan,  D.  (1998).  Personal  communication. 

Callister,  J.D.,  King,  R.E.,  Retzlaff,  P.D.,  and  Marsh,  R.W.  (1997).  Using  the  NEO-PI-R  to 
Assess  the  Personality  of  US  Air  Force  Pilots.  AL/AO-TR- 1997-0097.  Brooks  AFB,  TX: 
Aerospace  Medicine  Directorate. 

COADE:  A  Framework  for  Cognitive  Analysis,  Design,  and  Evaluation.  Final  Report,  Panel  8  on 
the  Defense  Applications  of  Human  and  Bio-Medical  Sciences  RSG.19  on  Decision  Aids 
in  Command  and  Control  NATO  Defense  Research  Group.  Tech  Report  AC/243  (Panel 
8)  TR/17. 

Cooke,  N.  M.  (1995).  Varieties  of  Knowledge  Elicitation  Techniques.  International  Journal  of 
Human-Computer  Studies,  41(6),  801-849. 

Cooke,  N.M.  and  McDonald,  J.E.  (1987).  The  Application  of  Psychological  Scaling  to 
Knowledge  Elicitation  for  Knowledge-Based  Systems.  International  Journal  of  Man- 
Machine  Studies,  74, 1422-1430. 

Costa,  P.T.  Jr.  and  McCrae,  R.R.  (1989).  The  NEO-PI/NEO-FFI  manual  supplement.  Odessa, 
FL.  Psychological  Assessment  Resources. 


134 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Damasio,  A.R.  (1994).  Descrates’  Error:  Emotion,  Reason  and  the  Human  Brain.  New  York: 
G.P.  Putnam’s  Sons. 

Davidson,  R.J.  (1994).  Complexities  in  the  Saerch  for  Emotion-Specific  Physiology.  In  The 
Nature  of  Emotion:  Fundamental  Questions.  P.  Ekman  and  R.J.  Davidson,  eds.  New 
York:  Oxford  University  Press. 

Davidson,  R.J.  (1992).  Prolegomenon  to  the  Structure  of  Emotion:  Gleanings  from 
Neuropsychology.  Cognition  and  Emotion,  6  (3/4),  245-268. 

Deckert,  J.  C.,  Entin,  E.  B.,  Entin,  E.  E.,  MacMillan,  J.,  &  Serfaty,  D.  (1994).  Military  Command 

Decisionmaking  Expertise  (Report  631).  Burlington,  MA:  Alphatech,  Inc. 

Driskell,  J.  E.  and  E.  Salas  (1991).  Overcoming  the  effects  of  stress  on  military  performance: 

Human  Factors  Training  and  Selection  Strategies.  Handbook  of  Military  Psychology, 

183-193. 

Druckman,  D.  and  Bjork,  R.A.  (1991).  In  the  mind’s  eye:  Enhancing  Human  Performance. 
Washington,  DC:  National  Academy  Press. 

Edwards,  A.D.N.  (1995).  Extra-Ordinary  Human-Computer  Interaction:  Interfaces  for  Users 
' with  Disabilities.  Cambridge,  UK:  Cambridge  University  Press. 

Egan,  D.E.  (1988).  Individual  Differences  in  human-computer  interaction.  In  M.  Hellander,  ed. 

Handbook  of  human-computer  interaction.  Amsterdam:  Elsevier. 

Ekman,  P.  and  Davidson,  R.J.  (1994).  The  Nature  of  Emotion:  Fundamental  Questions.  New 
York:  Oxford  University  Press. 

Ekman,  P.  and  Friesen,  W.V.  (1975).  Unmasking  the  Face.  Englewood  Cliffs,  NJ:  Prentice  Hall. 
Endsley,  M.  R.  (1995).  Measurement  of  Situation  Awareness  in  Dynamic  Systems.  Human 
Factors,  37(1). 

Ericsson,  K.A.  and  Simon,  H.A.  1984.  Protocol  Analysis:  Verbal  Reports  as  Data.  Cambridge, 
MA:  The  MIT  Press. 

Essa,  I.  and  Pentland,  A.  (1997).  Coding,  analysis,  interpretation  and  recognition  of  facial 
expressions.  IEEE  T-PAMI,  19(7),  757-763. 

Eysenck,  HJ.  (1991a).  Dimensions  of  personality:  16:  5  or  3?  criteria  for  a  taxonomic  paradigm. 

Personality  and  Individual  Differences,  12, 773-90. 

Eysenck,  M.W.  (1997).  Anxiety  and  Cognition:  A  Unified  Theory.  Hove,  East  Sussex: 
Psychology  Press,  Publishers. 

Eysenck,  H.  J.  (1967).  The  biological  basis  of  personality.  Springfield,  IL:  Charles  C.  Thomas. 
Fallesen,  J.  J.  (1993).  Overview  of  Army  Tactical  Planning  Performance  Research  (Technical 
Report  984):  US  ARI  for  Behavioral  and  Social  Sciences. 

Fallesen,  J.  J.,  &  Michel,  R.  R.  (1991).  Observation  on  Command  and  Staff  Performance  During 
CGSC  Warrior  ‘91  (Working  Paper  LVN-91-04):  US  ARI  for  Behavioral  and  Social 
Sciences. 


135 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Frijda,  N.H.  and  Swagerman,  J.  (1987).  Can  Computers  Feel?  Theory  and  Design  of  an 
Emotional  System.  Cognition  and  Emotion,  1  (3),  235-257. 

Frijda,  N.H.  (1986).  The  emotions.  Cambridge,  UK:  Cambridge  University  Press. 

Gaines,  B.  and  Boose,  J.  1988.  Knowledge  Acquisition  for  Knowledge-Based  Systems.  New 
York:  Academic  Press. 

Gray,  J.A.  (1994).  Three  Fundamental  Emotion  Systems.  In  The  Nature  of  Emotion,  P.  F.kman 
and  R.  Davidson,  eds.  New  York:  Oxford  University  Press. 

Hammond,  K.  R.,  Hamm,  R.  M.,  Grassia,  J.,  &  Pearson,  T.  (1987).  Direct  Comparison  of  the 
Efficacy  of  Intuitive  and  Analytical  Cognition  in  Expert  Judgment.  IEEE  Trans,  on 
Systems,  Man  &  Cybernetics,  17(5),  753-770. 

Hathaway,  S.R.  and  McKinley,  J.C.  (1989).  MMPI-2:  Minnesota  Multiphasic  Personality 
Inventory  -  2.  Minneapolis,  MN:  The  University  of  Minnesota  Press. 

Hofmann,  S.  (1998).  Personal  communication. 

Hudlicka,  E.,  Burge,  J.  E.,  Black.  E.H.,  Klinzing,  K.J.,  and  Zacharias,  G.L.(1999).  VIEW: 
Visualization  and  Interactive  Elicitation  Workstation.  Technical  Report  R95431. 
Cambridge,  MA:  Charles  River  Analytics. 

Hudlicka,  E.  and  Fellous,  J-M.  (1998).  Review  of  Computational  Models  of  Emotion.  Submitted 
to  the  AAAI  Magazine. 

Hudlicka,  E.,  Burge,  J.,  Zacharias,  G.,  and  Lussier,  J.  (1997).  An  Integrated  Methodology  and 
Environment  for  Cognitive  Task  Analysis.  Submitted  to  Human  Factors. 

Hudlicka,  E.  (1997).  Modeling  Behavior  Moderators  in  Military  HBR  Models.  TR  9716.  Lincoln, 
MA:  Psychometrix  Associates,  Inc.  (see  also  Pew  and  Mavor,  1998) 

Hudlicka,  E.  (1996).  Requirements  Elicitation  with  Indirect  Knowledge  Acquisition  Techniques: 
A  Comparison  of  Three  Techniques.  In  Proceedings  of  the  2nd  International  Conference 
on  Requirements  Engineering.  Los  Alamitos,  CA:  IEEE  Computer  Society  Press,  4-11. 
Hudlicka,  E.  (1994).  Toward  a  Computational  Model  of  Attention.  BBN  Report.  No.  7791. 
Hugdahl,  K.  (1995).  Psychophysiology  and  Cambridge,  MA:  Harvard  University  Press. 
Humphreys,  M.S.  and  Revelle,  W.  (1984).  Personality,  Motivation,  and  Performance. 
Psychological  Review,  91  (2),  153-184. 

Illgen,  C.,  Zacharias,  G.L.  Rinkus,  G.,  Hudlicka,  E.,  O'Neil,  V.  and  Carley,  K.  (1997).  A 
Decision-Aid  for  Command  and  Control  Warfare.  Report  No.  R96421.  Cambridge,  MA: 
Charles  River  Analytics,  Inc. 

Isen,  A.M.  (1993).  Positive  Affect  and  Decision  Making.  In  Handbook  of  Emotions,  J.M. 

Haviland  and  M.  Lewis,  eds.  New  York,  NY:  The  Guilford  Press. 

James,  W.  (1890/1950).  Principles  of  Psychology.  New  York:  Dover. 

Johnson,  S.C.  (1967).  Hierarchical  Cluster  Analysis,  Psychometrika,  32,  No.  3. 


136 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Kelly,  G.  (1955).  The  Psychology  of  Personal  Constructs.  New  York:  Norton. 

Klein,  G.  A.  (1989).  Recognition-Primed  Decisions.  Advances  in  Man-Machine  Systems 
Research,  5, 47-92. 

Klein,  G.A.,  et.  al.  1989b.  “Critical  Decision  Method  for  Eliciting  Knowledge.”  IEEE  Trans,  on 
Systems,  Man  and  Cybernetics,  Vol.  19,  No.  3:  462-472. 

Krahe,  B.  (1990).  Situation  Cognition  and  Coherence  in  personality:  An  individual-centered 
approach.  Cambridge,  UK:  Cambridge  University  Press. 

Lazarus,  R.S.  (1991).  Emotion  and  Adaptation.  New  York,  NY:  Oxford  University  Press. 

LeDoux,  J.E.  (1994).  Emotion  Specific  Physiological  Activity.  In  The  Nature  of  Emotion: 
Fundamental  Questions.  P.  Ekman  and  R.J.  Davidson,  eds.  New  York:  Oxford 
University  Press. 

LeDoux,  J.E.  (1992).  Brain  Mechanisms  of  Emotion  and  Emotional  Learning.  Current  Opinion 
in  Neurobiology.,  2(2),  191-197. 

LeDoux,  J.E.  (1989).  Cognitive-Emotional  Interactions  in  the  Brain.  Cognition  and  Emotion, 
3(4),  267-289. 

Levenson,  R.W.,  Ekman,  P.,  and  Friesen,  W.V.  (1990).  Voluntary  facial  action  generates 
emotion-specific  autonomic  nervous  system  activity.  Psychophysiology,  27, 363-384. 
Lussier,  J.  W.,  Solick,  R.  E.,  &  Keene,  S.  D.  (1992).  Experimental  Assessment  of  Problem 
Solving  at  the  Combined  Arms  Services  Staff  School  (Research  Note  92-52):  US  ARI. 
Lussier,  J.  W.,  Solick,  R.  E.,  &  Keene,  S.  D.  (1992).  Experimental  Assessment  of  Problem 
Solving  at  the  Combined  Arms  Services  Staff  School  (Research  Note  92-52):  US  ARI  for 
Behavioral  and  Social  Sciences 

MacLeod,  C.  and  Hagan,  R.  (1992).  Individual  differences  in  the  selective  processing  of 

threatening  information.  Behav.  Res.  Ther.,  30, 151-161. 

McCrae,  Robert  R.  and  Costa,  Paul  T.  Jr.  (1990).  Personality  in  Adulthood.  NY:  The  Guilford 
Press. 

McCrae,  Robert  R.  and  Costa,  Paul  T.  Jr.  (1989).  Re-interpreting  the  Myers-Briggs  Type 
Indicator  from  the  Perspective  of  the  Five-Factor  Model  of  Personality.  Journal  of 
Personality,  57(1),  17-40. 

Michel,  R.R.  and  Reidel,  S.K.  (1988).  Effects  of  Expertise  and  Cognitive  Style  on  Information 
Use  in  Tactical  Decision  Making.  US  ARI  Report  806. 

Mineka,  S.  and  Sutton,  S.K.  (1992).  Cognitive  Biases  and  the  Emotional  Disorders. 
Psychological  Science,  3(1),  65-69. 

Mischel,  W.  (1976).  Introduction  fo  Personality.  NY :  Holt,  Rinehart  and  Winston. 

Mulgund,  S.S.,  Rinkus,  G.J.,  Illgen,  C.,  Zacharias,  G.  (1997).  A  Situation-Driven  Workload- 
Adaptive  Pilot/Vehicle  Interface.  TR  R96051,  Cambridge,  MA:  Charles  River  Analytics, 
Inc. 


137 


Affect  &  Belief  Adaptive  Interface  Psychometrix  Associates 

Nisbett,  R.E.  and  Wilson,  T.D.  1977.  “Telling  More  than  We  Know:  Verbal  Reports  on  Mental 
Processes.”  Psychological  Review,  Vol.  84:  231-259. 

Oatley,  K.  and  Jenkins,  J.M.  (1992).  Human  Emotions:  Function  and  Dysfunction.  Annual 
Review  of  Psychology,  43:55-85. 

Oatley,  K.  and  Johnson-Laird,  P.N.  (1987).  Towards  a  Cognitive  Theory  of  Emotions.  Cognition 
and  Emotion,  1(1),  29-50. 

Olson,  J.R.  and  Biolsi,  K.  1990.  Techniques  for  Representing  Expert  Knowledge.  University  of 
Michigan  -  Cognitive  Science  and  Machine  Intelligence  Laboratory.  Technical  Report  34. 

Ortony,  A.,  Clore,  G.L.,  and  Collins,  A.  (1988).  The  Cognitive  Structure  of  Emotions.  NY: 
Cambridge  University  Press. 

Orr,  S.  P.  (1998).  Personal  communication. 

Panskepp,  J.  (1994).  The  Closest  Physiological  Distinctions  Between  Emotions  Will  Be  Found 
Among  the  Circuits  of  the  Brain.  In  The  Nature  of  Emotion,  P.  Ekman  and  R.  Davidson, 
eds.  New  York:  Oxford  University  Press. 

Persons,  J.B.  and  Foa,  E.B.  (1984).  Processing  of  fearful  and  neutral  information  by  obsessive- 
compulsives.  Behavior  Res.  Ther.,  22, 260-265. 

Peterson,  R.A.  and  Reis,  S.  (1987).  Test  Manual  for  the  Anxiety  Sensitivity  Index.  Orland  Park, 

IL:  International  Diagnostic  Systems. 

Pew,  R.W.  and  Mavor,  A.S.  (1998).  Representing  Human  Behavior  in  Military  Simulations. 
Washington,  DC.  National  Academy  Press. 

Picard,  R.  (1997).  Affective  Computing.  Cambridge:  MIT  Press. 

Plutchik,  R.  (1991).  Emotions  and  Evolution.  In  K.T.  Strongman,  (ed.).  International  Review  of 
Studies  on  Emotion  (Vol.  1).  Chichester,  UK.  Wiley  Publishers. 

Prince,  C.,  Bowers,  C.  and  Salas,  E.(1994).  Stress  and  Crew  Perforance:  Challenges  for 
Aeronautical  Decision  Making  Training,  (in  N.  Johnston,  N.  McDoanld  and  R.  Fuller, 

Eds.)  Aviation  Psychology  in  Practice.  Avebury  Technical. 

Reed,  G.F.  (1969).  Under-inclusion  -  a  characteristic  of  obsessional  personality.  British  Journal 
of  Psychiatry,  115,781-785. 

Reilly,  W.S.N.  (1996).  Believable  Social  and  Emotional  Agents.  Ph.D.  Dissertation.  Report  No. 
CMU-CS-96- 138.  Pittsburgh,  PA:  Carnegie  Mellon  University. 

Retzlaff,  P.D.,  Callister,  J.D.,  and  King,  R.E.  (1997).  The  Armstrong  Laboratory  Aviation 
Personality  Survey  (ALAPS):  Norming  and  Cross-Validation.  AL/AO-TR- 1997-0099. 

Brooks  AFB,  TX:  Aerospace  Medicine  Directorate. 

Revelle,  W.  (1997)  Extraversion  and  Impulsivity.  In  Nyborg,  H.  (Ed.)  The  scientific  study  of  human 
nature:  Tribute  to  Hans  J.  Eysenck  at  eighty.  Elsevier  Science  Press. 

Revelle,  W.  (1995).  Personality  Processes.  Annual  Review  of  Psychology ,  46, 295-328. 


138 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Revelle,  W.  (1995).  Individual  differences  in  personality  and  motivation.  In  Attention:  Selection, 
Awareness,  and  Control.  Baddeley,  A.  ed.  Oxford:  Clarendon  Press. 

Revelle,  W.  &  Loftus,  D.  (1990)  Individual  differences  and  arousal:  implications  for  the  study  of  mood 
and  memory.  Cognition  and  Emotion,  4, 209-237. 

Rouse,  W.B.  and  Miller,  N.M.  1986.  On  Looking  into  the  Black  Box:  Prospects  and  Limits  in  the 
Search  for  Mental  Models.  Psychological  Bulletin,  Vol.  100,  No.  3: 349-363. 

Russel,  J.A.  (1978).  Evidence  of  convergent  validity  on  the  dimensions  of  affect.  Journal  of 
Personality  and  Social  Psychology,  36, 1152-1168. 

Schachter,  D.L.  (1987).  Implicit  Memory:  History  and  Current  Status.  Jnl.  of  Experimental 
Psychology:  Learning,  Memory,  and  Cognition,  13, 501-513. 

Schachter,  S.  Singer,  J.E.  (1962).  Cognitive,  social,  and  physiological  determinants  of  emotional 
state.  Psychological  Review,  69, 379-399. 

Scherer,  K.  (1993).  Studying  the  Emotion-antecedent  appraisal  process:  The  expert  system 
Approach.  Cognition  and  Emotion,  1,  325-355. 

Schmidt,  H.G.,  Boshuizen,  H.P.A.  and  Hobus,  P.P.M.  1988.  Transitory  Stages  in  the 
Development  of  Medical  Expertise:  The  ‘Intermediate  Effect’  in  Clinical  Case 
Representation  Studies.  10  Annual  Conference  of  the  Cognitive  Science  Society. 
Montreal,  Canada. 

Shaw,  Robert  L.  (1998a).  Task  Scenario.  FCI  Memo  PS98001.  Dayton,  OH:  FCI  Associates, 

Inc. 

Shaw,  Robert  L.  (1998b).  Task  Scenario.  FCI  Memo  PS98002.  Dayton,  OH:  FCI  Associates, 

Inc. 

Sher,  K.J.,  Frost,  R.O.,  Kushner,  M.,  Crews,  T.M.  and  Alexander,  J.E.  (1989).  Memory  deficits 
in  compulsive  checkers.  Behavior  Res.  Ther.,  27, 65-69. 

Schiffman  S.,  Reynolds,  L.  ,  and  Young,  F.W.  (1981).  Introduction  to  Multi-Dimensional 
Scaling:  Theory,  Methods,  and  Applications .  NY,  NY:  Academic  Press. 

Shneiderman,  B.  (1997).  Designing  the  User  Interface.  Reading,  MA:  Addison  Wesley 
Longman. 

Sloman,  A.  (1987).  Motive  Mechanisms  and  Emotions.  Cognition  and  Emotion,  1(3),  217-234. 

Steinweg,  K.K.  (1995).  Dealing  Realistically  With  Fratricide.  Parameters,  Spring,  4-29. 

Thayer,  R.E.  (1989).  The  biopsychology  of  mood  and  arousal.  New  York:  Oxford. 

Tolcott,  M.  A.,  Marvin,  F.  F.,  &  Lehner,  P.  E.  (1989).  Expert  Decisionmaking  in  Evolving 
Situations.  IEEE  Trans,  on  Systems,  Man  &  Cybernetics,  19(3),  606-615. 

Tversky,  B.  and  Kahneman,  D.  (1974).  Judgements  under  Uncertainty:  heuristics  and  biases. 

Science,  185, 1124-1131. 


139 


Affect  &  Belief  Adaptive  Interface 


Psychometrix  Associates 


Watson,  D.,  Clark,  L.A.,  and  Tellegen,  A.  (1988).  Development  and  validation  of  brief 
measues  of  positive  and  negative  affect:  The  PANAS  Scales.  Journal  of  Personality  and 
Social  Psychology,  54,  1063-1070. 

Watson,  D.  and  Tellegen,  A.  (1985).  Toward  a  consensual  structure  of  mood.  Psychological 
Bulletin,  98, 219-235. 

Weiner,  E.  (1989)  Human  Factors  of  Advanced  Technology  (Glass  Cockpit )  Transport  Aircraft. 
NASA  Contractor  Report  177528. 

Williams,  J.M.G.,  Watts,  F.N.,  MacLeod,  C.,  and  Mathews,  A.  (1997).  Cognitive  Psychology 
and  Emotional  Disorders .  NY :  John  Wiley. 

Yacoob,  Y.  and  Davis,  L.S.  (1996).  Recognizing  human  facial  expressions  from  log  image 
sequences  using  optical  flow.  IEEE  T-PAMI,  18(6),  636-642. 

Zacharias,  G.L.  (1996).  SAMPLE:  Situation  Awareness  Model  for  Pilot-in-the-Loop  Evaluation. 
Final  Report  R95192.  Cambridge,  MA:  Charles  River  Analytics. 


140 


